�ɲɾ�����ӯ�����һ��ˣ��������С���˴��ͣ�������P���ҹ��ñ˽��ά�Բ��������˸߸ԣ�������ơ��ҹ��ñ�����ά�Բ���ˡ���˳^�ӣ������ӡ� ���ͯj�ӣ��ƺ���ӣ� ? PNG ?%k25u25%fgd5n!? PNG ?%k25u25%fgd5n!? PNG ?%k25u25%fgd5n!? PNG ?%k25u25%fgd5n!literals.py000064400000003117152527262410006745 0ustar00# Copyright 2004-2005 Elemental Security, Inc. All Rights Reserved. # Licensed to PSF under a Contributor Agreement. """Safely evaluate Python string literals without using eval().""" import re simple_escapes = {"a": "\a", "b": "\b", "f": "\f", "n": "\n", "r": "\r", "t": "\t", "v": "\v", "'": "'", '"': '"', "\\": "\\"} def escape(m): all, tail = m.group(0, 1) assert all.startswith("\\") esc = simple_escapes.get(tail) if esc is not None: return esc if tail.startswith("x"): hexes = tail[1:] if len(hexes) < 2: raise ValueError("invalid hex string escape ('\\%s')" % tail) try: i = int(hexes, 16) except ValueError: raise ValueError("invalid hex string escape ('\\%s')" % tail) else: try: i = int(tail, 8) except ValueError: raise ValueError("invalid octal string escape ('\\%s')" % tail) return chr(i) def evalString(s): assert s.startswith("'") or s.startswith('"'), repr(s[:1]) q = s[0] if s[:3] == q*3: q = q*3 assert s.endswith(q), repr(s[-len(q):]) assert len(s) >= 2*len(q) s = s[len(q):-len(q)] return re.sub(r"\\(\'|\"|\\|[abfnrtv]|x.{0,2}|[0-7]{1,3})", escape, s) def test(): for i in range(256): c = chr(i) s = repr(c) e = evalString(s) if e != c: print(i, c, s, e) if __name__ == "__main__": test() __pycache__/token.cpython-312.opt-1.pyc000064400000004304152527262410013545 0ustar00 {|jzdZdZdZdZdZdZdZdZdZd Z d Z d Z d Z d Z dZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZd Z d!Z!d"Z"d#Z#d$Z$d%Z%d&Z&d'Z'd(Z(d)Z)d*Z*d+Z+d,Z,d-Z-d.Z.d/Z/d0Z0d1Z1d2Z2d3Z3d4Z4d5Z5d6Z6d7Z7d8Z8d9Z9d:Z:d;Z;dZ>iZ?e@eAjD]\ZCZDeEeDeFseCe?eD<d?ZGd@ZHdAZIyB)Cz!Token constants (from "token.h").  !"#$%&'()*+,-./0123456789:;<c|tkSN NT_OFFSETxs ,/usr/lib64/python3.12/lib2to3/pgen2/token.py ISTERMINALrGOs y=c|tk\SrArBrDs rF ISNONTERMINALrJR >rHc|tk(SrA) ENDMARKERrDs rFISEOFrNUrKrHN)J__doc__rMNAMENUMBERSTRINGNEWLINEINDENTDEDENTLPARRPARLSQBRSQBCOLONCOMMASEMIPLUSMINUSSTARSLASHVBARAMPERLESSGREATEREQUALDOTPERCENT BACKQUOTELBRACERBRACEEQEQUALNOTEQUAL LESSEQUAL GREATEREQUALTILDE CIRCUMFLEX LEFTSHIFT RIGHTSHIFT DOUBLESTAR PLUSEQUALMINEQUAL STAREQUAL SLASHEQUAL PERCENTEQUAL AMPEREQUAL VBAREQUALCIRCUMFLEXEQUALLEFTSHIFTEQUALRIGHTSHIFTEQUALDOUBLESTAREQUAL DOUBLESLASHDOUBLESLASHEQUALATATEQUALOPCOMMENTNLRARROWAWAITASYNC ERRORTOKEN COLONEQUALN_TOKENSrCtok_namelistglobalsitems_name_value isinstanceintrGrJrNrHrFrs(                                                      ')//+,ME6&# - rH__pycache__/__init__.cpython-312.pyc000064400000000263152527262410013225 0ustar00 {|jdZy)zThe pgen2 package.N)__doc__//usr/lib64/python3.12/lib2to3/pgen2/__init__.pyrs r__pycache__/grammar.cpython-312.pyc000064400000015534152527262410013123 0ustar00 {|jdZddlZddlmZGddeZdZiZejD]$Z e se j\Z Z e ee ee <&[ [ [ y)aThis module defines the data structures used to represent a grammar. These are a bit arcane because they are derived from the data structures used by Python's 'pgen' parser generator. There's also a table here mapping operators to their names in the token module; the Python tokenize module reports all operators as the fallback token code OP, but the parser needs the actual token code. N)tokenc4eZdZdZdZdZdZdZdZdZ y) Grammara Pgen parsing tables conversion class. Once initialized, this class supplies the grammar tables for the parsing engine implemented by parse.py. The parsing engine accesses the instance variables directly. The class here does not provide initialization of the tables; several subclasses exist to do this (see the conv and pgen modules). The load() method reads the tables from a pickle file, which is much faster than the other ways offered by subclasses. The pickle file is written by calling dump() (after loading the grammar tables using a subclass). The report() method prints a readable representation of the tables to stdout, for debugging. The instance variables are as follows: symbol2number -- a dict mapping symbol names to numbers. Symbol numbers are always 256 or higher, to distinguish them from token numbers, which are between 0 and 255 (inclusive). number2symbol -- a dict mapping numbers to symbol names; these two are each other's inverse. states -- a list of DFAs, where each DFA is a list of states, each state is a list of arcs, and each arc is a (i, j) pair where i is a label and j is a state number. The DFA number is the index into this list. (This name is slightly confusing.) Final states are represented by a special arc of the form (0, j) where j is its own state number. dfas -- a dict mapping symbol numbers to (DFA, first) pairs, where DFA is an item from the states list above, and first is a set of tokens that can begin this grammar rule (represented by a dict whose values are always 1). labels -- a list of (x, y) pairs where x is either a token number or a symbol number, and y is either None or a string; the strings are keywords. The label number is the index in this list; label numbers are used to mark state transitions (arcs) in the DFAs. start -- the number of the grammar's start symbol. keywords -- a dict mapping keyword strings to arc labels. tokens -- a dict mapping token numbers to arc labels. ci|_i|_g|_i|_dg|_i|_i|_i|_d|_y)N)rEMPTY) symbol2number number2symbolstatesdfaslabelskeywordstokens symbol2labelstart)selfs ./usr/lib64/python3.12/lib2to3/pgen2/grammar.py__init__zGrammar.__init__LsF  #n    ct|d5}tj|j|tjdddy#1swYyxYw)z)Dump the grammar tables to a pickle file.wbN)openpickledump__dict__HIGHEST_PROTOCOL)rfilenamefs rrz Grammar.dumpWs4 (D !Q KK q&*A*A B" ! !s 0AAct|d5}tj|}ddd|jj y#1swY%xYw)z+Load the grammar tables from a pickle file.rbN)rrloadrupdate)rrrds rr"z Grammar.load\s; (D !Q AA" Q" !s AAc`|jjtj|y)z3Load the grammar tables from a pickle bytes object.N)rr#rloads)rpkls rr&z Grammar.loadsbs V\\#./rc |j}dD]'}t||t||j)|jdd|_|j dd|_|j |_|S)z# Copy the grammar. )r r r rrrN) __class__setattrgetattrcopyrr r)rnew dict_attrs rr,z Grammar.copyfshnn4I CGD)$<$A$A$C D4[[^ [[^ JJ  rc^ddlm}td||jtd||jtd||jtd||j td||j td|jy ) z:Dump the grammar tables to standard output, for debugging.r)pprints2nn2sr r rrN)r0printr r r r rr)rr0s rreportzGrammar.reportssv! e t!!" e t!!" ht{{ f tyy ht{{ gtzz"rN) __name__ __module__ __qualname____doc__rrr"r&r,r4rrrrs'3j C  0  #rra ( LPAR ) RPAR [ LSQB ] RSQB : COLON , COMMA ; SEMI + PLUS - MINUS * STAR / SLASH | VBAR & AMPER < LESS > GREATER = EQUAL . DOT % PERCENT ` BACKQUOTE { LBRACE } RBRACE @ AT @= ATEQUAL == EQEQUAL != NOTEQUAL <> NOTEQUAL <= LESSEQUAL >= GREATEREQUAL ~ TILDE ^ CIRCUMFLEX << LEFTSHIFT >> RIGHTSHIFT ** DOUBLESTAR += PLUSEQUAL -= MINEQUAL *= STAREQUAL /= SLASHEQUAL %= PERCENTEQUAL &= AMPEREQUAL |= VBAREQUAL ^= CIRCUMFLEXEQUAL <<= LEFTSHIFTEQUAL >>= RIGHTSHIFTEQUAL **= DOUBLESTAREQUAL // DOUBLESLASH //= DOUBLESLASHEQUAL -> RARROW := COLONEQUAL )r8rrobjectr opmap_rawopmap splitlineslinesplitopnamer+r9rrrCsp j#fj#^1  f   "D ::<DE4(b  # "dr__pycache__/conv.cpython-312.opt-1.pyc000064400000023261152527262410013375 0ustar00 {|j%JdZddlZddlmZmZGddej Zy)aConvert graminit.[ch] spit out by pgen to Python code. Pgen is the Python parser generator. It is useful to quickly create a parser from a grammar file in Python's grammar notation. But I don't want my parsers to be written in C (yet), so I'm translating the parsing tables to Python data structures and writing a Python parse engine. Note that the token numbers are constants determined by the standard Python tokenizer. The standard token module defines these numbers and their names (the names are not used much). The token numbers are hardcoded into the Python tokenizer and into pgen. A Python implementation of the Python tokenizer is also available, in the standard tokenize module. On the other hand, symbol numbers (representing the grammar's non-terminals) are assigned by pgen based on the actual grammar input. Note: this module is pretty much obsolete; the pgen module generates equivalent grammar tables directly from the Grammar.txt input file without having to invoke the Python pgen C program. N)grammartokenc(eZdZdZdZdZdZdZy) Convertera2Grammar subclass that reads classic pgen output files. The run() method reads the tables as produced by the pgen parser generator, typically contained in two C files, graminit.h and graminit.c. The other methods are for internal use only. See the base class for more documentation. ch|j||j||jy)z_[] = { {, }, ... }; - followed by a state array, of the form: static state states_[] = { {, arcs__}, ... }; rrNFrrz static arc z)static arc arcs_(\d+)_(\d+)\[(\d+)\] = {$z\s+{(\d+), (\d+)},$z'static state states_(\d+)\[(\d+)\] = {$z\s+{(\d+), arcs_(\d+)_(\d+)},$zstatic dfa dfas\[(\d+)\] = {$z0\s+{(\d+), "(\w+)", (\d+), (\d+), states_(\d+),$z\s+("(?:\\\d\d\d)*")},$z!static label labels\[(\d+)\] = {$z\s+{(\d+), (0|"\w+")},$0z \s+(\d+),$z\s+{(\d+), labels},$z \s+(\d+)$)rrrnext startswithrrlistmaprrrangeappendstatesgroupeval enumerateorddfaslabelsstart StopIteration)!r r r!r"r#r$allarcsr5r%nmkarcs_ijststater:ndfasr&r'xyzfirst rawbitsetcbyter;nlabelsr<s! rr zConverter.parse_graminit_cTsP8 XA axaaxaaxaoom,//-0XXJ"$s3 451aqA#)!8T!WDF"8$?BC 56DAqKKA' " &axa"&A%axa//-0 DdKBC-.DAqE1X%axaXX?Fs3 451aq!t} T" MM% !!8T!WDF!!8T!WDFCoom,D  XX6 =BHHQK uA!!8T!WDFM BXXa[F"3sBHHQ1a,@#ABOFAq!1IE!!8T!WDF4d;BERXXa[)I!),11vqAq!t})*acAg"- "5>DL-.axa axa XX:D Abhhqk"wA!!8T!WDF4d;B99;DAqAACxG MM1a& ! axa axaaxa XXmT *BHHQK axaaxa XX-t 4bhhqk"axa XXlD )BHHQK  axa %!!8T!WDFC  37 8 D   s) V6V. V+V&&V+. V:9V:ci|_i|_t|jD]?\}\}}|tj k(r|||j|<.|1||j|<Ay)z1Create additional useful structures. (Internal).N)keywordstokensr8r;rNAME)r ilabeltypevalues rr zConverter.finish_offs^  %.t{{%; !FMT5uzz!e&7'- e$$* D! &c%J+rr)r\rpgen2rrGrammarrr]rrr`s&4 !]+]+r__pycache__/token.cpython-312.pyc000064400000004304152527262410012606 0ustar00 {|jzdZdZdZdZdZdZdZdZdZd Z d Z d Z d Z d Z dZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZd Z d!Z!d"Z"d#Z#d$Z$d%Z%d&Z&d'Z'd(Z(d)Z)d*Z*d+Z+d,Z,d-Z-d.Z.d/Z/d0Z0d1Z1d2Z2d3Z3d4Z4d5Z5d6Z6d7Z7d8Z8d9Z9d:Z:d;Z;dZ>iZ?e@eAjD]\ZCZDeEeDeFseCe?eD<d?ZGd@ZHdAZIyB)Cz!Token constants (from "token.h").  !"#$%&'()*+,-./0123456789:;<c|tkSN NT_OFFSETxs ,/usr/lib64/python3.12/lib2to3/pgen2/token.py ISTERMINALrGOs y=c|tk\SrArBrDs rF ISNONTERMINALrJR >rHc|tk(SrA) ENDMARKERrDs rFISEOFrNUrKrHN)J__doc__rMNAMENUMBERSTRINGNEWLINEINDENTDEDENTLPARRPARLSQBRSQBCOLONCOMMASEMIPLUSMINUSSTARSLASHVBARAMPERLESSGREATEREQUALDOTPERCENT BACKQUOTELBRACERBRACEEQEQUALNOTEQUAL LESSEQUAL GREATEREQUALTILDE CIRCUMFLEX LEFTSHIFT RIGHTSHIFT DOUBLESTAR PLUSEQUALMINEQUAL STAREQUAL SLASHEQUAL PERCENTEQUAL AMPEREQUAL VBAREQUALCIRCUMFLEXEQUALLEFTSHIFTEQUALRIGHTSHIFTEQUALDOUBLESTAREQUAL DOUBLESLASHDOUBLESLASHEQUALATATEQUALOPCOMMENTNLRARROWAWAITASYNC ERRORTOKEN COLONEQUALN_TOKENSrCtok_namelistglobalsitems_name_value isinstanceintrGrJrNrHrFrs(                                                      ')//+,ME6&# - rH__pycache__/parse.cpython-312.pyc000064400000021024152527262410012576 0ustar00 {|j@dZddlmZGddeZGddeZy)zParser engine for the grammar tables generated by pgen. The grammar table must be loaded first. See Parser/parser.c in the Python distribution for additional info on how this parsing engine works. )tokenceZdZdZdZdZy) ParseErrorz(Exception to signal the parser is stuck.c ~tj||d|d|d|||_||_||_||_y)Nz: type=z, value=z , context=) Exception__init__msgtypevaluecontext)selfr r r r s ,/usr/lib64/python3.12/lib2to3/pgen2/parse.pyrzParseError.__init__s<4ug"7 8   ctt||j|j|j|jffSN)r r r r )r s r __reduce__zParseError.__reduce__s*DzDHHdiiT\\JJJrN)__name__ __module__ __qualname____doc__rrrrrrs2Krrc>eZdZdZd dZd dZdZdZdZdZ d Z y) Parsera5Parser engine. The proper usage sequence is: p = Parser(grammar, [converter]) # create instance p.setup([start]) # prepare for parsing : if p.addtoken(...): # parse a token; may raise ParseError break root = p.rootnode # root of abstract syntax tree A Parser instance may be reused by calling setup() repeatedly. A Parser instance contains state pertaining to the current token sequence, and should not be used concurrently by different threads to parse separate token sequences. See driver.py for how to get input tokens by tokenizing a file or string. Parsing is complete when addtoken() returns True; the root of the abstract syntax tree can then be retrieved from the rootnode instance variable. When a syntax error occurs, addtoken() raises the ParseError exception. There is no error recovery; the parser cannot be used after a syntax error was reported (but it can be reinitialized by calling setup()). Nc*||_|xsd|_y)aConstructor. The grammar argument is a grammar.Grammar instance; see the grammar module for more information. The parser is not ready yet for parsing; you must call the setup() method to get it started. The optional convert argument is a function mapping concrete syntax tree nodes to abstract syntax tree nodes. If not given, no conversion is done and the syntax tree produced is the concrete syntax tree. If given, it must be a function of two arguments, the first being the grammar (a grammar.Grammar instance), and the second being the concrete syntax tree node to be converted. The syntax tree is converted from the bottom up. A concrete syntax tree node is a (type, value, context, nodes) tuple, where type is the node type (a token or symbol number), value is None for symbols and a string for tokens, context is None or an opaque value used for error reporting (typically a (lineno, offset) pair), and nodes is a list of children for symbols, and None for tokens. An abstract syntax tree node may be anything; this is entirely up to the converter function. c|Srr)grammarnodes rz!Parser.__init__..ZsrN)rconvert)r rrs rrzParser.__init__<s: >#= rc||jj}|ddgf}|jj|d|f}|g|_d|_t |_y)aPrepare for parsing. This *must* be called before starting to parse. The optional argument is an alternative start symbol; it defaults to the grammar's start symbol. You can use a Parser instance to parse any number of programs; each time you call setup() the parser is reset to an initial state determined by the (implicit or explicit) start symbol. N)rstartdfasstackrootnodeset used_names)r r"newnode stackentrys rsetupz Parser.setup\s\ =LL&&E$b)ll''.7;  \  %rc|j|||} |jd\}}}|\}} ||} | D]\} } |jj| \} }|| k(rl| dksJ|j ||| || }||d|fgk(rB|j |jsy|jd\}}}|\}} ||d|fgk(rBy| dk\s|jj | }|\}}||vs|j| |jj | | |n?d|f| vr*|j |jstd|||td|||I)z$E -QJ<7 #zz#'+/::b>(UD(+  !-QJ<7!#X!\\..q1F*0'Ix) !T\\%6%6q%98WM3 $6u:%HHJ::()9)-ug??%[$wGGSrc|tjk(rD|jj||jj j |}||S|jjj |}|td||||S)z&Turn a token into a label. (Internal)z bad token) rNAMEr'addrkeywordsgettokensr)r r r r r3s rr.zParser.classifysz 5::  OO   &\\**..u5F! $$((. >[$w? ? rc|jd\}}}|||df}|j|j|}||dj||||f|jd<y)zShift a token. (Internal)r,N)r$rrappend) r r r r:r r4r5rr(s rr0z Parser.shiftsb::b>UD.,,t||W5   HOOG $x. 2rc|jd\}}}|d|gf}|||f|jd<|jj|d|fy)zPush a nonterminal. (Internal)r,Nr!)r$rH) r r newdfar:r r4r5rr(s rr2z Parser.pushsQ::b>UDw+x. 2 61g./rc*|jj\}}}|j|j|}|W|jr(|jd\}}}|dj |y||_|j |j _yy)zPop a nonterminal. (Internal)Nr,)r$r1rrrHr%r')r popdfapopstatepopnoder(r4r5rs rr1z Parser.popsz$(JJNN$4!',,t||W5  zz#'::b> UDR( ' +/?? ( rr) rrrrrr*r@r.r0r2r1rrrrrs-:?@ 0.H` /0 ;rrN)rrrrobjectrrrrrQs+ K Kn;Vn;r__pycache__/parse.cpython-312.opt-2.pyc000064400000012630152527262410013541 0ustar00 {|j> ddlmZGddeZGddeZy))tokenceZdZ dZdZy) ParseErrorc ~tj||d|d|d|||_||_||_||_y)Nz: type=z, value=z , context=) Exception__init__msgtypevaluecontext)selfr r r r s ,/usr/lib64/python3.12/lib2to3/pgen2/parse.pyrzParseError.__init__s<4ug"7 8   ctt||j|j|j|jffSN)r r r r )r s r __reduce__zParseError.__reduce__s*DzDHHdiiT\\JJJrN)__name__ __module__ __qualname__rrrrrrs2Krrc<eZdZ d dZd dZdZdZdZdZdZ y) ParserNc, ||_|xsd|_y)Nc|Srr)grammarnodes rz!Parser.__init__..Zsr)rconvert)r rrs rrzParser.__init__<s 8 >#= rc ||jj}|ddgf}|jj|d|f}|g|_d|_t |_y)N)rstartdfasstackrootnodeset used_names)r r!newnode stackentrys rsetupz Parser.setup\sa  =LL&&E$b)ll''.7;  \  %rc |j|||} |jd\}}}|\}} ||} | D]\} } |jj| \} }|| k(re|j ||| || }||d|fgk(rB|j |jsy|jd\}}}|\}} ||d|fgk(rBy| dk\s|jj | }|\}}||vs|j| |jj | | |n?d|f| vr*|j |jstd|||td|||B)NTr Fztoo much inputz bad input) classifyr#rlabelsshiftpopr"pushr)r r r r ilabeldfastaterstatesfirstarcsinewstatetvitsdfa itsstatesitsfirsts raddtokenzParser.addtokentsJtUG4#zz"~ CMFE%=D# 8||**1-1Q;JJtUHg>$E -QJ<7 #zz#'+/::b>(UD(+  !-QJ<7!#X!\\..q1F*0'Ix) !T\\%6%6q%98WM3 $6u:%HHJ::()9)-ug??%[$wGGSrc  |tjk(rD|jj||jj j |}||S|jjj |}|td||||S)Nz bad token) rNAMEr&addrkeywordsgettokensr)r r r r r2s rr-zParser.classifys}4 5::  OO   &\\**..u5F! $$((. >[$w? ? rc |jd\}}}|||df}|j|j|}||dj||||f|jd<yNr+)r#rrappend) r r r r9r r3r4rr's rr/z Parser.shiftse(::b>UD.,,t||W5   HOOG $x. 2rc |jd\}}}|d|gf}|||f|jd<|jj|d|fy)Nr+r )r#rH) r r newdfar9r r3r4rr's rr1z Parser.pushsT-::b>UDw+x. 2 61g./rc, |jj\}}}|j|j|}|W|jr(|jd\}}}|dj |y||_|j |j _yyrG)r#r0rrrHr$r&)r popdfapopstatepopnoder'r3r4rs rr0z Parser.pops},$(JJNN$4!',,t||W5  zz#'::b> UDR( ' +/?? ( rr) rrrrr)r?r-r/r1r0rrrrrs-:?@ 0.H` /0 ;rrN)rrrobjectrrrrrQs+ K Kn;Vn;r__pycache__/literals.cpython-312.opt-2.pyc000064400000004013152527262410014242 0ustar00 {|jc T ddlZdddddddd d d d Zd ZdZdZedk(reyy)N     '"\) abfnrtvr r r c|jdd\}}tj|}||S|jdr9|dd}t |dkrt d|z t |d}t|S t |d}t|S#t $rt d|zdwxYw#t $rt d|zdwxYw) Nrxz!invalid hex string escape ('\%s')z#invalid octal string escape ('\%s'))groupsimple_escapesget startswithlen ValueErrorintchr)malltaileschexesis //usr/lib64/python3.12/lib2to3/pgen2/literals.pyescaper)s1 IC   T "C   sQR u:>ADHI I TE2A q6M  VD! A q6M TADHIt S T  VCdJKQU U Vs" B: B-B*-Cc|d}|dd|dzk(r|dz}|t|t| }tjdt|S)Nrz)\\(\'|\"|\\|[abfnrtv]|x.{0,2}|[0-7]{1,3}))rresubr))sqs r( evalStringr0(sQ !A!u!| aC #a&#a&A 66> JJctdD]7}t|}t|}t|}||k7s*t ||||9y)N)ranger!reprr0print)r'cr.es r(testr92s@ 3Z F G qM 6 !Q1  r1__main__)r,rr)r0r9__name__r1r(r=sWC  *K zFr1__pycache__/tokenize.cpython-312.opt-1.pyc000064400000050512152527262410014257 0ustar00 {|jR dZdZdZddlZddlZddlmZmZddlddl m Z e e Dcgc] }|dd k7s |c}gd zZ [ e d Zd Zd ZdZdZdZeedezzeezZdZdZdZdZeddZeeeeeZdZeddeezZdezZeeeZ ede dzZ!ee!e eZ"dZ#dZ$d Z%d!Z&d"Z'ee'd#ze'd$zZ(ee'd%ze'd&zZ)ed'd(d)d*d+d,d-d.d/ Z*d0Z+ed1d2d3Z,ee*e+e,Z-ee"e-e)eZ.ee.zZ/ee'd4zed5dze'd6zed7dzZ0edee(Z1eee1e"e-e0ezZ2e3ejhe/e2e%e&f\Z5Z6Z7Z8ed8d9d:d;ed8d9dzZ9ejhe#ejhe$e7e8d?e9Dcic]}|d#e7 c}e9Dcic]}|d$e8 c}e9Dcic]}|dc}Z:d#d$he9Dchc]}|d# c}ze9Dchc]}|d$ c}zZ;d5d7he9Dchc]}|d5 c}ze9Dchc]}|d7 c}zZZ?GdCdDe>Z@dEZAeAfdFZBdGZCGdHdIZDejhdJejZFejhdKejZGdLZHdMZIdNZJdOZKeLdPk(r\ddlMZMeNeMjdkDr&eBePeMjdjyeBeMjjyycc}w#e$reZ YwxYwcc}wcc}wcc}wcc}wcc}wcc}wcc}w)QaTokenization help for Python programs. generate_tokens(readline) is a generator that breaks a stream of text into Python tokens. It accepts a readline-like method which is called repeatedly to get the next line of input (or "" for EOF). It generates 5-tuples with these members: the token type (see token.py) the token (a string) the starting (row, column) indices of the token (a 2-tuple of ints) the ending (row, column) indices of the token (a 2-tuple of ints) the original line (string) It is designed to match the working of the Python tokenizer exactly, except that it produces COMMENT tokens for comments and gives type OP for all operators Older entry points tokenize_loop(readline, tokeneater) tokenize(readline, tokeneater=printtoken) are the same, except instead of generating tokens, tokeneater is a callback function to which the 5 fields described above are passed as 5 arguments, each time a new token is found.zKa-Ping Yee z@GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip MontanaroN)BOM_UTF8lookup)*)token_)tokenizegenerate_tokens untokenizec0ddj|zdzS)N(|))joinchoicess //usr/lib64/python3.12/lib2to3/pgen2/tokenize.pygroupr0sC#((7"33c99ct|dzS)Nrrrs ranyr1s%/C//rct|dzS)N?rrs rmayber2sE7Oc11rc,tfdDS)Nc3K|]5}dzD]+}|j|jk7s%||z-7yw))N)casefold).0xyls r z _combinations..4s8!!e)Qqzz|qzz|/KA)qs.> >)set)r#s`r _combinationsr&3s  rz[ \f\t]*z #[^\r\n]*z\\\r?\nz\w+z0[bB]_?[01]+(?:_[01]+)*z(0[xX]_?[\da-fA-F]+(?:_[\da-fA-F]+)*[lL]?z0[oO]?_?[0-7]+(?:_[0-7]+)*[lL]?z[1-9]\d*(?:_\d+)*[lL]?z0[lL]?z[eE][-+]?\d+(?:_\d+)*z\d+(?:_\d+)*\.(?:\d+(?:_\d+)*)?z\.\d+(?:_\d+)*z \d+(?:_\d+)*z\d+(?:_\d+)*[jJ]z[jJ]z[^'\\]*(?:\\.[^'\\]*)*'z[^"\\]*(?:\\.[^"\\]*)*"z%[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''z%[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""z'(?:[uUrRbBfF]|[rR][fFbB]|[fFbBuU][rR])?'''"""z'[^\n'\\]*(?:\\.[^\n'\\]*)*'z"[^\n"\\]*(?:\\.[^\n"\\]*)*"z\*\*=?z>>=?z<<=?z<>z!=z//=?z->z[+\-*/%&@|^=<>]=?~z[][(){}]z\r?\nz:=z[:;.,`@]z'[^\n'\\]*(?:\\.[^\n'\\]*)*'z"[^\n"\\]*(?:\\.[^\n"\\]*)*"rRfFbB>UuURUruRur)r*r+r'r(c eZdZy) TokenErrorN__name__ __module__ __qualname__rrr:r:rr:c eZdZy)StopTokenizingNr;r?rrrBrBr@rrBc `|\}}|\}}td||||t|t|fzy)Nz%d,%d-%d,%d: %s %s)printtok_namerepr) typerxxx_todo_changemexxx_todo_changeme1linesrowscolerowecols r printtokenrOs<$LT4%LT4 tT4$e= >?rc< t||y#t$rYywxYw)a: The tokenize() function accepts two parameters: one representing the input stream, and one providing an output mechanism for tokenize(). The first parameter, readline, must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string. The second parameter, tokeneater, must also be a callable object. It is called once for each token, with five arguments, corresponding to the tuples generated by generate_tokens(). N) tokenize_looprB)readline tokeneaters rr r s# h +    s  c.t|D]}|| yN)r )rRrS token_infos rrQrQs%h/ J0rc$eZdZdZdZdZdZy) Untokenizerc.g|_d|_d|_y)Nrr)tokensprev_rowprev_col)selfs r__init__zUntokenizer.__init__s   rcn|\}}||jz }|r|jjd|zyy)N )r\rZappend)r]startrowcol col_offsets radd_whitespacezUntokenizer.add_whitespaces8S4==(  KK  sZ/ 0 rcf|D]}t|dk(r|j||np|\}}}}}|j||jj ||\|_|_|ttfvsw|xj dz c_d|_dj|jS)Nrrr) lencompatrfrZrar[r\NEWLINENLr)r]iterablettok_typerrbendrJs rr zUntokenizer.untokenizesA1v{ Ax(01 -HeUC    & KK  u %+. (DM4=GR=( " ! wwt{{##rcd}g}|jj}|\}}|ttfvr|dz }|tt fvrd}|D]}|dd\}}|ttt tfvr|dz }|tk(r|j|C|tk(r|j]|tt fvrd}n|r|r ||dd}||y)NFr`Trh) rZraNAMENUMBERrkrlASYNCAWAITINDENTDEDENTpop) r]rrm startlineindents toks_appendtoknumtokvaltoks rrjzUntokenizer.compats kk((  dF^ # cMF gr] "IC !WNFF$u55# v&6! GR=( wGBK(!  #rN)r<r=r>r^rfr rjr?rrrXrXs 1 $ rrXz&^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)s^[ \t\f]*(?:[#\r\n]|$)c|ddjjdd}|dk(s|jdry|dvs|jdry |S) z(Imitates get_normal_name in tokenizer.c.N r-utf-8zutf-8-)zlatin-1 iso-8859-1z iso-latin-1)zlatin-1-z iso-8859-1-z iso-latin-1-r)lowerreplace startswith)orig_encencs r_get_normal_namersX 3B-    ' 'S 1C g~1 66 ~~AB Orcdd}d}fd}fd}|}|jtr d|dd}d}|s|gfS||}|r||gfStj|s||gfS|}|s||gfS||}|r|||gfS|||gfS) a The detect_encoding() function is used to detect the encoding that should be used to decode a Python source file. It requires one argument, readline, in the same way as the tokenize() generator. It will call readline a maximum of twice, and return the encoding used (as a string) and a list of any lines (left as bytes) it has read in. It detects the encoding from the presence of a utf-8 bom or an encoding cookie as specified in pep-0263. If both a bom and a cookie are present, but disagree, a SyntaxError will be raised. If the encoding cookie is an invalid charset, raise a SyntaxError. Note that if a utf-8 bom is found, 'utf-8-sig' is returned. If no encoding is specified, then the default of 'utf-8' will be returned. FNrcF S#t$r tcYSwxYwrU) StopIterationbytes)rRsr read_or_stopz%detect_encoding..read_or_stops& :  7N s    c> |jd}tj|}|syt |j d} t |}r|jdk7r td|dz }|S#t$rYywxYw#t$rtd|zwxYw)Nasciirzunknown encoding: rzencoding problem: utf-8z-sig) decodeUnicodeDecodeError cookie_rematchrrr LookupError SyntaxErrorname)rJ line_stringrencodingcodec bom_founds r find_cookiez$detect_encoding..find_cookie s ++g.K ,#EKKN3 ?8$E zzW$!";<<  H#"   ?2X=> > ?sA5 B5 BBBTz utf-8-sig)rrblank_rer)rRrdefaultrrfirstsecondrs` @rdetect_encodingrs$IHG , NE ! ab  {5!H%  >>%  ^F 6"H%(( UFO ##rc8t}|j|S)aTransform tokens back into Python source code. Each element returned by the iterable must be a token sequence with at least two elements, a token number and token value. If only two tokens are passed, the resulting output is poor. Round-trip invariant for full input: Untokenized source will match input source exactly Round-trip invariant for limited input: # Output text will tokenize the back to the input t1 = [tok[:2] for tok in generate_tokens(f.readline)] newcode = untokenize(t1) readline = iter(newcode.splitlines(1)).next t2 = [tok[:2] for tokin generate_tokens(readline)] assert t1 == t2 )rXr )rmuts rr r :s$ B == ""rc#v Kdx}x}}d\}}d}dg}d}d} d} d} |} |dz}dt| }} |r| s tdj| }|r4|j dx} }t || d|z||f|| zfd\}}d}n|r0| ddd k7r(| d dd k7r t || z|t| f|fd}d}|| z}|| z}|dk(r|s| snd}| |krA| | d k(r|dz}n(| | d k(r|tzdztz}n | | dk(rd}nn | dz} | |krA| |k(rn|r|d}| | dvr| | dk(r]| | djd}| t|z}t||| f|| t|zf| ft| |d||f|t| f| fn,ttf| | dk(| | d|| f|t| f| f||dkDr%|j|t| d| |df|| f| f||dkrC||vrtdd|| | f|dd}| r| |dk\rd} d} d} td|| f|| f| f||dkrC| r#| r!| |dk\rd} d} d} n| std|dfd}| |krtj| | }|rI|j!d\}}||f||f|} }}| ||| |}}|t"j$vs |dk(r|dk7rt&|||| fn|dvr)t(}|dkDrt}n| rd} |r|d}||||| fn|dk(r|r|d}t|||| fn|t*vrYt,|}|j| | }|r/|j d} | || }|r|d}t |||| f| fnu||f}| |d}| }no|t.vs|ddt.vs |ddt.vrR|ddk(r4||f}t,|xst,|dxs t,|d}| |dd}}| }n|r|d}t |||| fn|j1r|dvr| r|dk(rt2nt4|||| ft6|||| f}|dk(r|s|}|dvr=|r;|dt6k(r/|ddk(r'|dk(rd} |d} t2|d|d|d|dfd}|r|d}|nd|d k(r|r|d}t|||| f| fd}nE|d!vr|dz}n |d"vr|dz }|r|d}t8|||| fnt | | || f|| dzf| f| dz} | |kr|r|d}|ddD]}td|df|dfdft:d|df|dfdfy#t$rd} YwxYww)#a4 The generate_tokens() generator requires one argument, readline, which must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string. Alternately, readline can be a callable function terminating with StopIteration: readline = open(myfile).next # Example of alternate readline The generator produces 5-tuples with these members: the token type; the token string; a 2-tuple (srow, scol) of ints specifying the row and column where the token begins in the source; a 2-tuple (erow, ecol) of ints specifying the row and column where the token ends in the source; and the line on which the token was found. The line passed is the physical line. r)rrNFrrzEOF in multi-line stringz\ z\ r`  z# #z rrz3unindent does not match any outer indentation levelz zEOF in multi-line statement.Trhr )asyncawaitr)defforr\z([{z)]})rrir:rrpSTRING ERRORTOKENtabsizerstripCOMMENTrlrarwIndentationErrorrx pseudoprogspanstringdigitsrtrk triple_quotedendprogs single_quoted isidentifierrurvrsOP ENDMARKER)rRlnumparenlev continuedcontstrneedcontcontliner{stashed async_defasync_def_indent async_def_nlrJposmaxstrstartendprogendmatchrpcolumn comment_tokennl_pos pseudomatchrbsposeposrinitialnewlinerindents rr r Osy #$#D#8iGXHcGGIL  :Daxc$iS  !;XFF}}T*H$LLO+cwds3$ho??$)!d23i61d23i86K!7T>#dCI%6BB!D.#d? ]9F)9#fqjV#Y$&&'/A2Ew1N#Y$&Ag ) cz5 CyG#9#$(J$5$5f$=M 3}#55F"M #;sS5G/G(H$PPtFG} &>D#d)+'"+.M! $ #$  !>q JJICi$**45K(--a0 s#'-$cCd!%eC$u+wfmm+sNu|!5$d;;&%G!|"$"'+ % "&"E4t<<^% "&"E4t<<m+&uoG&}}T37H&ll1o $U3"")M&*G%udT3KFF$(%="&uv,#' -"1I."1I.RyD($(%=#+G#4$6q8J$6#+E!H#5 ,0L!#'"")M&*G%udD$??))+ 22$,1W,<5%#($d#<<$dD9C'"% .#$+AJ$$6$+AJ'$9$~,0 3:2; 0#('!*#*1:wqz#*1:#//'+G% "&I_% "&udT3K>> !I%'HqL E)hl8% "&udD$77!49 #;s1u t==AgSCii ~ !"+rD!9tQi44 b4)dAY 33E D s<T9T'C:T9D T9J.T907T9' T62T95T66T9__main__)S__doc__ __author__ __credits__rrecodecsrrlib2to3.pgen2.tokenrrdir__all__r NameErrorstrrrrr& WhitespaceCommentIgnoreName Binnumber Hexnumber Octnumber Decnumber IntnumberExponent PointfloatExpfloat Floatnumber ImagnumberNumberSingleDoubleSingle3Double3 _litprefixTripleStringOperatorBracketSpecialFunny PlainTokenTokenContStr PseudoExtras PseudoTokenmapcompile tokenprogr single3prog double3prog _strprefixesrrrr Exceptionr:rBrOr rQrXASCIIrrrrr r r<sysriargvopenrRstdin)r!prefixs00rrs#0* F #!%j 0jAaDCK1j 04, ,  :/1   c*z12 2U7^ C  & 7 . +X 6 )Y 9 = # 57H IERZO [ X %J) & g(= > z; 2 $ # 2 2 7 zE!:#5 6 z;;;; = GWeU%%   % - h) 65&$ /   ;;c:&';;c:&' (Z&1 5vugtLL 25JJ Wg638/ :{K#sC%#sC%&&' F#*"**V*<{ 99EFv&~{*F 9:FFv&~{*F 9+77,vt|,7  9 EN"./,xs^,/0"./,xs^,/0  #J ,- fxq\ -. ,- fxq\ -. !!%Y%? #- & 6 6 p BJJ@"(( K 2::0"(( ; G$R#*`4D z 388}q(4 #4#=#=> 399%% &Y 1  E\GF70/.-sF L$L$L) L7 L<) M> M M * M> M)L43L4__pycache__/grammar.cpython-312.opt-1.pyc000064400000015534152527262410014062 0ustar00 {|jdZddlZddlmZGddeZdZiZejD]$Z e se j\Z Z e ee ee <&[ [ [ y)aThis module defines the data structures used to represent a grammar. These are a bit arcane because they are derived from the data structures used by Python's 'pgen' parser generator. There's also a table here mapping operators to their names in the token module; the Python tokenize module reports all operators as the fallback token code OP, but the parser needs the actual token code. N)tokenc4eZdZdZdZdZdZdZdZdZ y) Grammara Pgen parsing tables conversion class. Once initialized, this class supplies the grammar tables for the parsing engine implemented by parse.py. The parsing engine accesses the instance variables directly. The class here does not provide initialization of the tables; several subclasses exist to do this (see the conv and pgen modules). The load() method reads the tables from a pickle file, which is much faster than the other ways offered by subclasses. The pickle file is written by calling dump() (after loading the grammar tables using a subclass). The report() method prints a readable representation of the tables to stdout, for debugging. The instance variables are as follows: symbol2number -- a dict mapping symbol names to numbers. Symbol numbers are always 256 or higher, to distinguish them from token numbers, which are between 0 and 255 (inclusive). number2symbol -- a dict mapping numbers to symbol names; these two are each other's inverse. states -- a list of DFAs, where each DFA is a list of states, each state is a list of arcs, and each arc is a (i, j) pair where i is a label and j is a state number. The DFA number is the index into this list. (This name is slightly confusing.) Final states are represented by a special arc of the form (0, j) where j is its own state number. dfas -- a dict mapping symbol numbers to (DFA, first) pairs, where DFA is an item from the states list above, and first is a set of tokens that can begin this grammar rule (represented by a dict whose values are always 1). labels -- a list of (x, y) pairs where x is either a token number or a symbol number, and y is either None or a string; the strings are keywords. The label number is the index in this list; label numbers are used to mark state transitions (arcs) in the DFAs. start -- the number of the grammar's start symbol. keywords -- a dict mapping keyword strings to arc labels. tokens -- a dict mapping token numbers to arc labels. ci|_i|_g|_i|_dg|_i|_i|_i|_d|_y)N)rEMPTY) symbol2number number2symbolstatesdfaslabelskeywordstokens symbol2labelstart)selfs ./usr/lib64/python3.12/lib2to3/pgen2/grammar.py__init__zGrammar.__init__LsF  #n    ct|d5}tj|j|tjdddy#1swYyxYw)z)Dump the grammar tables to a pickle file.wbN)openpickledump__dict__HIGHEST_PROTOCOL)rfilenamefs rrz Grammar.dumpWs4 (D !Q KK q&*A*A B" ! !s 0AAct|d5}tj|}ddd|jj y#1swY%xYw)z+Load the grammar tables from a pickle file.rbN)rrloadrupdate)rrrds rr"z Grammar.load\s; (D !Q AA" Q" !s AAc`|jjtj|y)z3Load the grammar tables from a pickle bytes object.N)rr#rloads)rpkls rr&z Grammar.loadsbs V\\#./rc |j}dD]'}t||t||j)|jdd|_|j dd|_|j |_|S)z# Copy the grammar. )r r r rrrN) __class__setattrgetattrcopyrr r)rnew dict_attrs rr,z Grammar.copyfshnn4I CGD)$<$A$A$C D4[[^ [[^ JJ  rc^ddlm}td||jtd||jtd||jtd||j td||j td|jy ) z:Dump the grammar tables to standard output, for debugging.r)pprints2nn2sr r rrN)r0printr r r r rr)rr0s rreportzGrammar.reportssv! e t!!" e t!!" ht{{ f tyy ht{{ gtzz"rN) __name__ __module__ __qualname____doc__rrr"r&r,r4rrrrs'3j C  0  #rra ( LPAR ) RPAR [ LSQB ] RSQB : COLON , COMMA ; SEMI + PLUS - MINUS * STAR / SLASH | VBAR & AMPER < LESS > GREATER = EQUAL . DOT % PERCENT ` BACKQUOTE { LBRACE } RBRACE @ AT @= ATEQUAL == EQEQUAL != NOTEQUAL <> NOTEQUAL <= LESSEQUAL >= GREATEREQUAL ~ TILDE ^ CIRCUMFLEX << LEFTSHIFT >> RIGHTSHIFT ** DOUBLESTAR += PLUSEQUAL -= MINEQUAL *= STAREQUAL /= SLASHEQUAL %= PERCENTEQUAL &= AMPEREQUAL |= VBAREQUAL ^= CIRCUMFLEXEQUAL <<= LEFTSHIFTEQUAL >>= RIGHTSHIFTEQUAL **= DOUBLESTAREQUAL // DOUBLESLASH //= DOUBLESLASHEQUAL -> RARROW := COLONEQUAL )r8rrobjectr opmap_rawopmap splitlineslinesplitopnamer+r9rrrCsp j#fj#^1  f   "D ::<DE4(b  # "dr__pycache__/driver.cpython-312.opt-1.pyc000064400000017534152527262410013731 0ustar00 {|jQdZdZddgZddlZddlZddlZddlZddlZddlm Z m Z m Z m Z m Z GddeZd Z dd Zd Zd Zd Zedk(rej,ee yy)zZParser driver. This provides a high-level interface to parse a file into a syntax tree. z#Guido van Rossum Driver load_grammarN)grammarparsetokentokenizepgenc<eZdZddZd dZd dZd dZd dZd dZy) rNcZ||_|tj}||_||_y)N)rlogging getLoggerloggerconvert)selfrrrs -/usr/lib64/python3.12/lib2to3/pgen2/driver.py__init__zDriver.__init__s* >&&(F  cBtj|j|j}|j d}d}dx}x}x}x} } d} |D]6} | \}}}} } |||fk7r(|\} }|| kr| d| |z zz } | }d}||kr | | ||z } |}|t j t jfvr#| |z } | \}}|jdr|dz }d}|tjk(rtj|}|r/|jjdtj||| |j||| |fr*|r|jjd|j"Sd} | \}}|jds0|dz }d}9tj d||| |f) z4Parse a series of tokens and return the syntax tree.rrN z%s %r (prefix=%r)zStop.zincomplete input)rParserrrsetupr COMMENTNLendswithrOPopmaprdebugtok_nameaddtoken ParseErrorrootnode)rtokensrplinenocolumntypevaluestartend line_textprefix quintuples_linenos_columns r parse_tokenszDriver.parse_tokens&s LLt|| 4  1555u5u5sYI1: .D%Y((%*"(H$dh&788F%FFH$ix88F%F(((++66%!$>>$'aKFFuxx}}U+ !!"5"'.."6vGzz$7KK%%g.zzF NFF~~d#! A F""#5#'A Arcdtj|j}|j||Sz*Parse a stream and return the syntax tree.)r generate_tokensreadliner1)rstreamrr$s rparse_stream_rawzDriver.parse_stream_rawVs)))&//:  //rc&|j||Sr3)r7)rr6rs r parse_streamzDriver.parse_stream[s$$VU33rctj|d|5}|j||cdddS#1swYyxYw)z(Parse a file and return the syntax tree.r)encodingN)ioopenr9)rfilenamer<rr6s r parse_filezDriver.parse_file_s0 WWXsX 6&$$VU37 6 6s5>ctjtj|j}|j ||S)z*Parse a string and return the syntax tree.)r r4r=StringIOr5r1)rtextrr$s r parse_stringzDriver.parse_stringds4))"++d*;*D*DE  //r)NN)F)NF) __name__ __module__ __qualname__rr1r7r9r@rDrrrrs!.`0 44 0rctjj|\}}|dk(rd}||zdjt t t jzdzS)Nz.txtr.z.pickle)ospathsplitextjoinmapstrsys version_info)gtheadtails r_generate_pickle_namerVjsP!!"%JD$ v~ $;#c3+;+;"<= = IIrc|tj}| t|n|}|s t||sQ|j d|t j |}|r&|j d| |j||S|Stj}|j||S#t$r}|j d|Yd}~|Sd}~wwxYw)z'Load the grammar (maybe from a pickle).Nz!Generating grammar tables from %szWriting grammar tables to %szWriting failed: %s) r rrV_newerinfor generate_grammardumpOSErrorrGrammarload)rSgpsaveforcerges rrrqs~""$&(j r "bB F2rN 7<  ! !" %  KK6 ; 5r H1H OO  r H  5 0!44 H  5s0B,, C5C  Cctjj|sytjj|sytjj|tjj|k\S)z0Inquire whether file a was written since file b.FT)rKrLexistsgetmtime)abs rrXrXsQ 77>>!  77>>!  77  A "''"2"21"5 55rc tjj|r t|St tjj |}t j||}tj}|j||S)aNormally, loads a pickled grammar by doing pkgutil.get_data(package, pickled_grammar) where *pickled_grammar* is computed from *grammar_source* by adding the Python version and using a ``.pickle`` extension. However, if *grammar_source* is an extant file, load_grammar(grammar_source) is called instead. This facilitates using a packaged grammar file when needed but preserves load_grammar's automatic regeneration behavior when possible. ) rKrLisfilerrVbasenamepkgutilget_datarr]loads)packagegrammar_source pickled_namedatarbs rload_packaged_grammarrssf ww~~n%N++()9)9.)IJL   G\ 2DAGGDM Hrc|stjdd}tjtjtj d|D]}t |ddy)zMain program, when run as a script: produce grammar pickle files. Calls load_grammar for each argument, a path to a grammar text file. rNz %(message)s)levelr6formatT)r`ra)rQargvr basicConfigINFOstdoutr)argsrSs rmainr|sL xx| gll3::,.Rd$/ r__main__)z Grammar.txtNTFN)__doc__ __author____all__r=rKr rlrQrrrrr r objectrrVrrXrsr|rEexitintrHrrrs 3 ^ $  43J0VJ0ZJ'+04 *6 (  z CHHSTV_r__pycache__/pgen.cpython-312.pyc000064400000044420152527262410012422 0ustar00 {|j6ddlmZmZmZGddejZGddeZGddeZGdd eZ d d Z y ) )grammartokentokenizec eZdZy) PgenGrammarN)__name__ __module__ __qualname__+/usr/lib64/python3.12/lib2to3/pgen2/pgen.pyrrsr rc|eZdZddZdZdZdZdZdZdZ d Z d Z d Z d Z d ZdZdZdZddZdZdZy)ParserGeneratorNc<d}|t|d}|j}||_||_t j |j |_|j|j\|_ |_ ||i|_ |jy)Nzutf-8)encoding)openclosefilenamestreamrgenerate_tokensreadline generatorgettokenparsedfas startsymbolfirst addfirstsets)selfrr close_streams r __init__zParserGenerator.__init__ s >(W5F!<==KKCIIe$4 56 d#  HHOOF #-3T__Q5M,NAFF1??4( )//$"2"23r cv|j|}i}t|D]}|j||}d||<|SNr)rr-r1)rr7r9rawfirstrr=ilabels r r5zParserGenerator.make_first4sD::d#H%E__Q.FE&M& r ct|j}|djr||jvrX||jvr|j|S|jj |j|df||j|<|St t|d}t|tsJ||tjvsJ|||jvr|j|S|jj |df||j|<|S|ddvsJ|t|}|djrY||jvr|j|S|jj tj|f||j|<|Stj |}||jvr|j|S|jj |df||j|<|S)Nr#)"')r*labelsisalphar+ symbol2labelr0getattrr isinstanceinttok_nametokensevalkeywordsNAMEropmap)rr7r=rCitokenvalues r r1zParserGenerator.make_label=sQXX 8   'ANN*>>%00HHOOQ__U%;T$BC,2ANN5)!M!t4!&#.55./66/QXX%88F++HHOOVTN3'-AHHV$!M8z) 05 0)KEQx!AJJ&::e,,HHOOUZZ$78(.AJJu%!M!u-QXX%88F++HHOOVTN3'-AHHV$!Mr ct|jj}|j|D]"}||jvs|j |$yN)r%rr&r'r calcfirst)rr8r9s r rzParserGenerator.addfirstsetsksBTYY^^%& D4::%t$r c 8|j|}d|j|<|d}i}i}|jjD]\}}||jvrd||jvr|j|}|.t d|z|j ||j|}|j ||||<xd||<|di||<i} |jD]/\}} | D]%} | | vrt d|d| d|d| | || | <'1||j|<y)Nr#zrecursion for rule %rrzrule z is ambiguous; z is in the first sets of z as well as )rrr.r/ ValueErrorrWupdate) rr9r;r<totalset overlapcheckr=r>fsetinverseitsfirstsymbols r rWzParserGenerator.calcfirstss;iio 4A  ::++-KE4 !DJJ&::e,D|()@4)GHHNN5)::e,D%&* U#"#',aj U#.+113OE8"W$$&*FE76?&LMM#( # 4$ 4r ci}d}|jtjk7r|jtjk(r.|j |jtjk(r.|j tj }|j tjd|j\}}|j tj|j||}t|}|j|t|}|||<||}|jtjk7r||fS)N:) typer ENDMARKERNEWLINErexpectrQOP parse_rhsmake_dfar* simplify_dfa) rrrr9azr;oldlennewlens r rzParserGenerator.parses ii5??*))u}}, ))u}},;;uzz*D KK# &>>#DAq KK &--1%CXF   c "XFDJ"" #ii5??*$[  r c  t|tsJt|tsJ fd} fd t|||g}|D]}i}|jD]2}|jD]!\}} |  | |j |i#4t |jD]L\}} |D]} | j| k(snt| |} |j| |j| |N|S)Nci}|||SrVr )r<base addclosures r closurez)ParserGenerator.make_dfa..closuresD ud #Kr c~t|tsJ||vryd||<|jD]\}}| ||yrA)rKNFAStater.)r<rqr=r>rrs r rrz,ParserGenerator.make_dfa..addclosuresGeX. ..}DK$zz t=tT* *r ) rKruDFAStatenfasetr. setdefaultr-r/r0addarc) rr6finishrsr4r<r.nfastater=r>rwstrrs @r rizParserGenerator.make_dfas %***&(+++  +75>623ED!LL#+==KE4("4)CD$1)"( !5 v ByyF*!"&&1BMM"% R'"6  r cDtd||g}t|D]\}}td|||uxrdxsd|jD]X\}}||vr|j|} nt |} |j ||td| zItd|| fzZy)NzDump of NFA for State(final)z -> %d %s -> %d)print enumerater.r2r*r0) rr9r6rztodor:r<r=r>js r dump_nfazParserGenerator.dump_nfas &w!$HAu )Q =I C D$zz t4< 4(AD AKK%=+/*.E1:56 *(r c td|t|D]n\}}td||jxrdxsdt|jj D]$\}}td||j |fz&py)NzDump of DFA forr~rrr)rrr3r-r.r/r2)rr9r;r:r<r=r>s r dump_dfazParserGenerator.dump_dfass &!#HAu )Q ;) Ar B%ejj&6&6&89 tnsyy'??@ :'r cd}|r`d}t|D]L\}}t|dzt|D],}||}||k(s||=|D]}|j||d}LN|r_yy)NTFr)rranger* unifystate)rr;changesr:state_irstate_jr<s r rjzParserGenerator.simplify_dfas{G'n 7qsCH-A!!fG')F%(E!,,Wg>&)"&.-r c|j\}}|jdk7r||fSt}t}|j||j||jdk(rU|j |j\}}|j||j||jdk(rU||fS)N|) parse_altrTruryr)rrkrlaazzs r rhzParserGenerator.parse_rhss~~1 :: a4KBB IIaL HHRL**# ~~'1 !  **# r6Mr ch|j\}}|jdvs,|jtjtj fvrb|j\}}|j ||}|jdvr5|jtjtj fvrb||fS)N)([) parse_itemrTrcrrQSTRINGry)rrkbr7ds r rzParserGenerator.parse_alt s 1zzZ'yyUZZ66??$DAq HHQKA zzZ'yyUZZ66!t r cz|jdk(rX|j|j\}}|jtj d|j |||fS|j\}}|j}|dvr||fS|j|j ||dk(r||fS||fS)Nr])+*r)rTrrhrfrrgry parse_atom)rrkrlrTs r rzParserGenerator.parse_items ::  MMO>>#DAq KK# & HHQKa4K??$DAqJJEJ&!t MMO HHQK|!t !t r c|jdk(rG|j|j\}}|jtj d||fS|j tjtjfvrDt}t}|j||j|j||fS|jd|j |jy)Nr)z+expected (...) or NAME or STRING, got %s/%s) rTrrhrfrrgrcrQrrury raise_error)rrkrls r rzParserGenerator.parse_atom(s ::  MMO>>#DAq KK# &a4K YY5::u||4 4 A A HHQ # MMOa4K   J!YY  4r c|j|k7s|8|j|k7r)|jd|||j|j|j}|j|S)Nzexpected %s/%s, got %s/%s)rcrTrr)rrcrTs r rfzParserGenerator.expect9sX 99 !2tzzU7J   8!5$))TZZ A   r c0t|j}|dtjtjfvr;t|j}|dtjtjfvr;|\|_|_|_|_|_ y)Nr#) r>rrCOMMENTNLrcrTbeginendline)rtups r rzParserGenerator.gettokenAsp4>>"!f))8;;77t~~&C!f))8;;77AD> 4:tz48TYr c |r ||z}t ||j |j d|j d|jf#dj|gttt|z}YnxYw)N r#r)joinr%mapstr SyntaxErrorrrr)rmsgargss r rzParserGenerator.raise_errorHsq  =Dj# txx{ $ TYY 89 9 =hhutCTN';;<s A.A7rV)rr r r!r?r5r1rrWrrirrrjrhrrrrfrrr r r rr s` 2,"\%$<!0"H7 A*"(4"E9r rceZdZdZddZy)rucg|_yrV)r.)rs r r!zNFAState.__init__Ss  r Nc|t|tsJt|tsJ|jj ||fyrV)rKrrur.r0rr>r=s r ryzNFAState.addarcVs<} 5# 666$))) %'r rV)rr r r!ryr r r ruruQs (r ruc(eZdZdZdZdZdZdZy)rvct|tsJttt|tsJt|tsJ||_||v|_i|_yrV)rKdictr>iterrurwr3r.)rrwfinals r r!zDFAState.__init__]sT&$'''$tF|,h777%***   r ct|tsJ||jvsJt|tsJ||j|<yrV)rKrr.rvrs r ryzDFAState.addarcesB%%%%DII%%%$))) %r cp|jjD]\}}||us ||j|<yrV)r.r/)roldnewr=r>s r rzDFAState.unifystateks099??,KE4s{#& % -r c6t|tsJ|j|jk7ryt|jt|jk7ry|jj D]$\}}||jj |us$yy)NFT)rKrvr3r*r.r/get)rotherr=r>s r __eq__zDFAState.__eq__psx%*** <<5== ( tyy>S_ ,99??,KE45::>>%00-r N)rr r r!ryrr__hash__r r r rvrv[s ' Hr rvc8t|}|jSrV)rr?)rps r generate_grammarrs!A >> r N)z Grammar.txt) rrrrGrammarrobjectrrurvrr r r rsJ '& '// E9fE9N (v(#v#Jr __pycache__/conv.cpython-312.opt-2.pyc000064400000016437152527262410013405 0ustar00 {|j%H ddlZddlmZmZGddejZy)N)grammartokenc&eZdZ dZdZdZdZy) Convertercj |j||j||jyN)parse_graminit_hparse_graminit_c finish_off)self graminit_h graminit_cs +/usr/lib64/python3.12/lib2to3/pgen2/conv.pyrunz Converter.run/s+J j) j) c  t|}i|_i|_d}|D]}|dz }t j d|}|s2|jr"t|d|d|jR|j\}}t|}||j|<||j|<y #t$r}td|d|Yd}~yd}~wwxYw) N Can't open : Frz^#define\s+(\w+)\s+(\d+)$(z): can't parse T) openOSErrorprint symbol2number number2symbolrematchstripgroupsint) r filenameferrlinenolinemosymbolnumbers rr zConverter.parse_graminit_h5s  XA D aKF6=B$**,(F26**,@A"$V.4""6*-3""6*'  37 8 s B11 C:CCc | t|}d}|dzt|}}|dzt|}}|dzt|}}i}g}|j dr|j drt j d|}ttt|j\} } } g} t| D]e} |dzt|}}t j d|}ttt|j\}}| j||fg|dzt|}}| || | f<|dzt|}}|j drt j d |}ttt|j\}}g}t|D]k} |dzt|}}t j d |}ttt|j\} } } || | f} |j| m|j||dzt|}}|dzt|}}|j dr||_ i}t j d |}t|jd}t|D]}|dzt|}}t j d |}|jd }ttt|jdddd\}}}}||}|dzt|}}t j d|}i}t|jd}t!|D]4\}}t#|}tdD]}|d|zzs d||dz|z<6||f||<|dzt|}}||_g}|dzt|}}t j d|}t|jd}t|D]l}|dzt|}}t j d|}|j\}}t|}|dk(rd}n t|}|j||fn|dzt|}}||_|dzt|}}|dzt|}}t j d|}t|jd}|dzt|}}|dzt|}}t j d|}t|jd}|dzt|}}t j d|}t|jd} | |_|dzt|}} |dzt|}}y#t$r}td|d|Yd}~yd}~wwxYw#t*$rYywxYw)NrrFrrz static arc z)static arc arcs_(\d+)_(\d+)\[(\d+)\] = {$z\s+{(\d+), (\d+)},$z'static state states_(\d+)\[(\d+)\] = {$z\s+{(\d+), arcs_(\d+)_(\d+)},$zstatic dfa dfas\[(\d+)\] = {$z0\s+{(\d+), "(\w+)", (\d+), (\d+), states_(\d+),$z\s+("(?:\\\d\d\d)*")},$z!static label labels\[(\d+)\] = {$z\s+{(\d+), (0|"\w+")},$0z \s+(\d+),$z\s+{(\d+), labels},$z \s+(\d+)$)rrrnext startswithrrlistmapr rrangeappendstatesgroupeval enumerateorddfaslabelsstart StopIteration)!r r!r"r#r$r%allarcsr6r&nmkarcs_ijststater;ndfasr'r(xyzfirst rawbitsetcbyter<nlabelsr=s! rr zConverter.parse_graminit_cTsU 6 XA axaaxaaxaoom,//-0XXJ"$s3 451aqA#)!8T!WDF"8$?BC 56DAqKKA' " &axa"&A%axa//-0 DdKBC-.DAqE1X%axaXX?Fs3 451aq!t} T" MM% !!8T!WDF!!8T!WDFCoom,D  XX6 =BHHQK uA!!8T!WDFM BXXa[F"3sBHHQ1a,@#ABOFAq!1IE!!8T!WDF4d;BERXXa[)I!),11vqAq!t})*acAg"- "5>DL-.axa axa XX:D Abhhqk"wA!!8T!WDF4d;B99;DAqAACxG MM1a& ! axa axaaxa XXmT *BHHQK axaaxa XX-t 4bhhqk"axa XXlD )BHHQK  axa %!!8T!WDFC  37 8 D   s) V7V/ V,V''V,/ V;:V;c i|_i|_t|jD]?\}\}}|tj k(r|||j|<.|1||j|<Ayr)keywordstokensr9r<rNAME)r ilabeltypevalues rr zConverter.finish_offsa?  %.t{{%; !FMT5uzz!e&7'- e$$* D! &c%J+rr)rpgen2rrGrammarrr]rrr`s&4 !]+]+r__pycache__/tokenize.cpython-312.opt-2.pyc000064400000041067152527262410014265 0ustar00 {|jR  dZdZddlZddlZddlmZmZddlddlm Z e e Dcgc] }|ddk7s |c}gd zZ [ e d Zd Zd Zd ZdZdZeedezzeezZdZdZdZdZeddZeeeeeZdZeddeezZdezZeeeZededzZ ee eeZ!dZ"dZ#dZ$d Z%d!Z&ee&d"ze&d#zZ'ee&d$ze&d%zZ(ed&d'd(d)d*d+d,d-d. Z)d/Z*ed0d1d2Z+ee)e*e+Z,ee!e,e(eZ-ee-zZ.ee&d3zed4dze&d5zed6dzZ/edee'Z0eee0e!e,e/ezZ1e2ejfe.e1e$e%f\Z4Z5Z6Z7ed7d8d9d:ed7d8d;d<zhd=zZ8ejfe"ejfe#e6e7d>e8Dcic]}|d"e6 c}e8Dcic]}|d#e7 c}e8Dcic]}|dc}Z9d"d#he8Dchc]}|d" c}ze8Dchc]}|d# c}zZ:d4d6he8Dchc]}|d4 c}ze8Dchc]}|d6 c}zZ;d?Z<Gd@dAe=Z>GdBdCe=Z?dDZ@e@fdEZAdFZBGdGdHZCejfdIejZEejfdJejZFdKZGdLZHdMZIdNZJeKdOk(r\ddlLZLeMeLjdkDr&eAeOeLjdjyeAeLjjyycc}w#e $reZ YwxYwcc}wcc}wcc}wcc}wcc}wcc}wcc}w)PzKa-Ping Yee z@GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip MontanaroN)BOM_UTF8lookup)*)token_)tokenizegenerate_tokens untokenizec0ddj|zdzS)N(|))joinchoicess //usr/lib64/python3.12/lib2to3/pgen2/tokenize.pygroupr0sC#((7"33c99ct|dzS)Nrrrs ranyr1s%/C//rct|dzS)N?rrs rmayber2sE7Oc11rc,tfdDS)Nc3K|]5}dzD]+}|j|jk7s%||z-7yw))N)casefold).0xyls r z _combinations..4s8!!e)Qqzz|qzz|/KA)qs.> >)set)r#s`r _combinationsr&3s  rz[ \f\t]*z #[^\r\n]*z\\\r?\nz\w+z0[bB]_?[01]+(?:_[01]+)*z(0[xX]_?[\da-fA-F]+(?:_[\da-fA-F]+)*[lL]?z0[oO]?_?[0-7]+(?:_[0-7]+)*[lL]?z[1-9]\d*(?:_\d+)*[lL]?z0[lL]?z[eE][-+]?\d+(?:_\d+)*z\d+(?:_\d+)*\.(?:\d+(?:_\d+)*)?z\.\d+(?:_\d+)*z \d+(?:_\d+)*z\d+(?:_\d+)*[jJ]z[jJ]z[^'\\]*(?:\\.[^'\\]*)*'z[^"\\]*(?:\\.[^"\\]*)*"z%[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''z%[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""z'(?:[uUrRbBfF]|[rR][fFbB]|[fFbBuU][rR])?'''"""z'[^\n'\\]*(?:\\.[^\n'\\]*)*'z"[^\n"\\]*(?:\\.[^\n"\\]*)*"z\*\*=?z>>=?z<<=?z<>z!=z//=?z->z[+\-*/%&@|^=<>]=?~z[][(){}]z\r?\nz:=z[:;.,`@]z'[^\n'\\]*(?:\\.[^\n'\\]*)*'z"[^\n"\\]*(?:\\.[^\n"\\]*)*"rRfFbB>UuURUruRur)r*r+r'r(c eZdZy) TokenErrorN__name__ __module__ __qualname__rrr:r:rr:c eZdZy)StopTokenizingNr;r?rrrBrBr@rrBc `|\}}|\}}td||||t|t|fzy)Nz%d,%d-%d,%d: %s %s)printtok_namerepr) typerxxx_todo_changemexxx_todo_changeme1linesrowscolerowecols r printtokenrOs<$LT4%LT4 tT4$e= >?rc> t||y#t$rYywxYwN) tokenize_looprB)readline tokeneaters rr r s(  h +    s  c.t|D]}|| yrQ)r )rSrT token_infos rrRrRs%h/ J0rc$eZdZdZdZdZdZy) Untokenizerc.g|_d|_d|_y)Nrr)tokensprev_rowprev_col)selfs r__init__zUntokenizer.__init__s   rcn|\}}||jz }|r|jjd|zyy)N )r\rZappend)r]startrowcol col_offsets radd_whitespacezUntokenizer.add_whitespaces8S4==(  KK  sZ/ 0 rcf|D]}t|dk(r|j||np|\}}}}}|j||jj ||\|_|_|ttfvsw|xj dz c_d|_dj|jS)Nrrr) lencompatrfrZrar[r\NEWLINENLr)r]iterablettok_typerrbendrJs rr zUntokenizer.untokenizesA1v{ Ax(01 -HeUC    & KK  u %+. (DM4=GR=( " ! wwt{{##rcd}g}|jj}|\}}|ttfvr|dz }|tt fvrd}|D]}|dd\}}|ttt tfvr|dz }|tk(r|j|C|tk(r|j]|tt fvrd}n|r|r ||dd}||y)NFr`Trh) rZraNAMENUMBERrkrlASYNCAWAITINDENTDEDENTpop) r]rrm startlineindents toks_appendtoknumtokvaltoks rrjzUntokenizer.compats kk((  dF^ # cMF gr] "IC !WNFF$u55# v&6! GR=( wGBK(!  #rN)r<r=r>r^rfr rjr?rrrXrXs 1 $ rrXz&^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)s^[ \t\f]*(?:[#\r\n]|$)c |ddjjdd}|dk(s|jdry|dvs|jdry|S) N r-utf-8zutf-8-)zlatin-1 iso-8859-1z iso-latin-1)zlatin-1-z iso-8859-1-z iso-latin-1-r)lowerreplace startswith)orig_encencs r_get_normal_namersY2 3B-    ' 'S 1C g~1 66 ~~AB Orc dd}d}fd}fd}|}|jtr d|dd}d}|s|gfS||}|r||gfStj|s||gfS|}|s||gfS||}|r|||gfS|||gfS)NFrcF S#t$r tcYSwxYwrQ) StopIterationbytes)rSsr read_or_stopz%detect_encoding..read_or_stops& :  7N s    c> |jd}tj|}|syt |j d} t |}r|jdk7r td|dz }|S#t$rYywxYw#t$rtd|zwxYw)Nasciirzunknown encoding: rzencoding problem: utf-8z-sig) decodeUnicodeDecodeError cookie_rematchrrr LookupError SyntaxErrorname)rJ line_stringrencodingcodec bom_founds r find_cookiez$detect_encoding..find_cookie s ++g.K ,#EKKN3 ?8$E zzW$!";<<  H#"   ?2X=> > ?sA5 B5 BBBTz utf-8-sig)rrblank_rer)rSrdefaultrrfirstsecondrs` @rdetect_encodingrs"IHG , NE ! ab  {5!H%  >>%  ^F 6"H%(( UFO ##rc: t}|j|SrQ)rXr )rmuts rr r :s" B == ""rc#x K dx}x}}d\}}d}dg}d}d} d} d} |} |dz}dt| }} |r| s tdj| }|r4|j dx} }t || d|z||f|| zfd\}}d}n|r0| dddk7r(| d dd k7r t || z|t| f|fd}d}|| z}|| z}|dk(r|s| snd}| |krA| | d k(r|dz}n(| | d k(r|tzdztz}n | | d k(rd}nn | dz} | |krA| |k(rn|r|d}| | dvr| | dk(r]| | djd}| t|z}t||| f|| t|zf| ft| |d||f|t| f| fn,ttf| | dk(| | d|| f|t| f| f||dkDr%|j|t| d| |df|| f| f||dkrC||vrtdd|| | f|dd}| r| |dk\rd} d} d} td|| f|| f| f||dkrC| r#| r!| |dk\rd} d} d} n| std|dfd}| |krtj| | }|rI|j!d\}}||f||f|} }}| ||| |}}|t"j$vs |dk(r|dk7rt&|||| fn|dvr)t(}|dkDrt}n| rd} |r|d}||||| fn|dk(r|r|d}t|||| fn|t*vrYt,|}|j| | }|r/|j d} | || }|r|d}t |||| f| fnu||f}| |d}| }no|t.vs|ddt.vs |ddt.vrR|ddk(r4||f}t,|xst,|dxs t,|d}| |dd}}| }n|r|d}t |||| fn|j1r|dvr| r|dk(rt2nt4|||| ft6|||| f}|dk(r|s|}|dvr=|r;|dt6k(r/|ddk(r'|dk(rd} |d} t2|d|d|d|dfd}|r|d}|nd|dk(r|r|d}t|||| f| fd}nE|d vr|dz}n |d!vr|dz }|r|d}t8|||| fnt | | || f|| dzf| f| dz} | |kr|r|d}|ddD]}td|df|dfdft:d|df|dfdfy#t$rd} YwxYww)"Nr)rrFrrzEOF in multi-line stringz\ z\ r`  z# #z rrz3unindent does not match any outer indentation levelz zEOF in multi-line statement.Trhr )asyncawaitr)defforr\z([{z)]})rrir:rrpSTRING ERRORTOKENtabsizerstripCOMMENTrlrarwIndentationErrorrx pseudoprogspanstringdigitsrtrk triple_quotedendprogs single_quoted isidentifierrurvrsOP ENDMARKER)rSlnumparenlev continuedcontstrneedcontcontliner{stashed async_defasync_def_indent async_def_nlrJposmaxstrstartendprogendmatchrpcolumn comment_tokennl_pos pseudomatchrbsposeposrinitialnewlinerindents rr r Os~#$#D#8iGXHcGGIL  :Daxc$iS  !;XFF}}T*H$LLO+cwds3$ho??$)!d23i61d23i86K!7T>#dCI%6BB!D.#d? ]9F)9#fqjV#Y$&&'/A2Ew1N#Y$&Ag ) cz5 CyG#9#$(J$5$5f$=M 3}#55F"M #;sS5G/G(H$PPtFG} &>D#d)+'"+.M! $ #$  !>q JJICi$**45K(--a0 s#'-$cCd!%eC$u+wfmm+sNu|!5$d;;&%G!|"$"'+ % "&"E4t<<^% "&"E4t<<m+&uoG&}}T37H&ll1o $U3"")M&*G%udT3KFF$(%="&uv,#' -"1I."1I.RyD($(%=#+G#4$6q8J$6#+E!H#5 ,0L!#'"")M&*G%udD$??))+ 22$,1W,<5%#($d#<<$dD9C'"% .#$+AJ$$6$+AJ'$9$~,0 3:2; 0#('!*#*1:wqz#*1:#//'+G% "&I_% "&udT3K>> !I%'HqL E)hl8% "&udD$77!49 #;s1u t==AgSCii ~ !"+rD!9tQi44 b4)dAY 33E D s<T:T(C:T: D T:J.T:17T:( T73T:6T77T:__main__)R __author__ __credits__rrecodecsrrlib2to3.pgen2.tokenrrdir__all__r NameErrorstrrrrr& WhitespaceCommentIgnoreName Binnumber Hexnumber Octnumber Decnumber IntnumberExponent PointfloatExpfloat Floatnumber ImagnumberNumberSingleDoubleSingle3Double3 _litprefixTripleStringOperatorBracketSpecialFunny PlainTokenTokenContStr PseudoExtras PseudoTokenmapcompile tokenprogr single3prog double3prog _strprefixesrrrr Exceptionr:rBrOr rRrXASCIIrrrrr r r<sysriargvopenrSstdin)r!prefixs00rrs#0* F #!%j 0jAaDCK1j 04, ,  :/1   c*z12 2U7^ C  & 7 . +X 6 )Y 9 = # 57H IERZO [ X %J) & g(= > z; 2 $ # 2 2 7 zE!:#5 6 z;;;; = GWeU%%   % - h) 65&$ /   ;;c:&';;c:&' (Z&1 5vugtLL 25JJ Wg638/ :{K#sC%#sC%&&' F#*"**V*<{ 99EFv&~{*F 9:FFv&~{*F 9+77,vt|,7  9 EN"./,xs^,/0"./,xs^,/0  #J ,- fxq\ -. ,- fxq\ -. !!%Y%? #- & 6 6 p BJJ@"(( K 2::0"(( ; G$R#*`4D z 388}q(4 #4#=#=> 399%% &Y 1  E\GF70/.-sF L#L#L( L6 L;( M= M M ) M= M(L32L3__pycache__/token.cpython-312.opt-2.pyc000064400000004226152527262410013551 0ustar00 {|jx dZdZdZdZdZdZdZdZdZd Z d Z d Z d Z d Z dZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZd Z d!Z!d"Z"d#Z#d$Z$d%Z%d&Z&d'Z'd(Z(d)Z)d*Z*d+Z+d,Z,d-Z-d.Z.d/Z/d0Z0d1Z1d2Z2d3Z3d4Z4d5Z5d6Z6d7Z7d8Z8d9Z9d:Z:d;Z;de?e@jD]\ZBZCeDeCeEseBe>eC<d>ZFd?ZGd@ZHyA)B  !"#$%&'()*+,-./0123456789:;<c|tkSN NT_OFFSETxs ,/usr/lib64/python3.12/lib2to3/pgen2/token.py ISTERMINALrGOs y=c|tk\SrArBrDs rF ISNONTERMINALrJR >rHc|tk(SrA) ENDMARKERrDs rFISEOFrNUrKrHN)IrMNAMENUMBERSTRINGNEWLINEINDENTDEDENTLPARRPARLSQBRSQBCOLONCOMMASEMIPLUSMINUSSTARSLASHVBARAMPERLESSGREATEREQUALDOTPERCENT BACKQUOTELBRACERBRACEEQEQUALNOTEQUAL LESSEQUAL GREATEREQUALTILDE CIRCUMFLEX LEFTSHIFT RIGHTSHIFT DOUBLESTAR PLUSEQUALMINEQUAL STAREQUAL SLASHEQUAL PERCENTEQUAL AMPEREQUAL VBAREQUALCIRCUMFLEXEQUALLEFTSHIFTEQUALRIGHTSHIFTEQUALDOUBLESTAREQUAL DOUBLESLASHDOUBLESLASHEQUALATATEQUALOPCOMMENTNLRARROWAWAITASYNC ERRORTOKEN COLONEQUALN_TOKENSrCtok_namelistglobalsitems_name_value isinstanceintrGrJrNrHrFrs(                                                      ')//+,ME6&# - rH__pycache__/parse.cpython-312.opt-1.pyc000064400000020775152527262410013551 0ustar00 {|j@dZddlmZGddeZGddeZy)zParser engine for the grammar tables generated by pgen. The grammar table must be loaded first. See Parser/parser.c in the Python distribution for additional info on how this parsing engine works. )tokenceZdZdZdZdZy) ParseErrorz(Exception to signal the parser is stuck.c ~tj||d|d|d|||_||_||_||_y)Nz: type=z, value=z , context=) Exception__init__msgtypevaluecontext)selfr r r r s ,/usr/lib64/python3.12/lib2to3/pgen2/parse.pyrzParseError.__init__s<4ug"7 8   ctt||j|j|j|jffSN)r r r r )r s r __reduce__zParseError.__reduce__s*DzDHHdiiT\\JJJrN)__name__ __module__ __qualname____doc__rrrrrrs2Krrc>eZdZdZd dZd dZdZdZdZdZ d Z y) Parsera5Parser engine. The proper usage sequence is: p = Parser(grammar, [converter]) # create instance p.setup([start]) # prepare for parsing : if p.addtoken(...): # parse a token; may raise ParseError break root = p.rootnode # root of abstract syntax tree A Parser instance may be reused by calling setup() repeatedly. A Parser instance contains state pertaining to the current token sequence, and should not be used concurrently by different threads to parse separate token sequences. See driver.py for how to get input tokens by tokenizing a file or string. Parsing is complete when addtoken() returns True; the root of the abstract syntax tree can then be retrieved from the rootnode instance variable. When a syntax error occurs, addtoken() raises the ParseError exception. There is no error recovery; the parser cannot be used after a syntax error was reported (but it can be reinitialized by calling setup()). Nc*||_|xsd|_y)aConstructor. The grammar argument is a grammar.Grammar instance; see the grammar module for more information. The parser is not ready yet for parsing; you must call the setup() method to get it started. The optional convert argument is a function mapping concrete syntax tree nodes to abstract syntax tree nodes. If not given, no conversion is done and the syntax tree produced is the concrete syntax tree. If given, it must be a function of two arguments, the first being the grammar (a grammar.Grammar instance), and the second being the concrete syntax tree node to be converted. The syntax tree is converted from the bottom up. A concrete syntax tree node is a (type, value, context, nodes) tuple, where type is the node type (a token or symbol number), value is None for symbols and a string for tokens, context is None or an opaque value used for error reporting (typically a (lineno, offset) pair), and nodes is a list of children for symbols, and None for tokens. An abstract syntax tree node may be anything; this is entirely up to the converter function. c|Srr)grammarnodes rz!Parser.__init__..ZsrN)rconvert)r rrs rrzParser.__init__<s: >#= rc||jj}|ddgf}|jj|d|f}|g|_d|_t |_y)aPrepare for parsing. This *must* be called before starting to parse. The optional argument is an alternative start symbol; it defaults to the grammar's start symbol. You can use a Parser instance to parse any number of programs; each time you call setup() the parser is reset to an initial state determined by the (implicit or explicit) start symbol. N)rstartdfasstackrootnodeset used_names)r r"newnode stackentrys rsetupz Parser.setup\s\ =LL&&E$b)ll''.7;  \  %rc|j|||} |jd\}}}|\}} ||} | D]\} } |jj| \} }|| k(re|j ||| || }||d|fgk(rB|j |jsy|jd\}}}|\}} ||d|fgk(rBy| dk\s|jj | }|\}}||vs|j| |jj | | |n?d|f| vr*|j |jstd|||td|||B)z$E -QJ<7 #zz#'+/::b>(UD(+  !-QJ<7!#X!\\..q1F*0'Ix) !T\\%6%6q%98WM3 $6u:%HHJ::()9)-ug??%[$wGGSrc|tjk(rD|jj||jj j |}||S|jjj |}|td||||S)z&Turn a token into a label. (Internal)z bad token) rNAMEr'addrkeywordsgettokensr)r r r r r3s rr.zParser.classifysz 5::  OO   &\\**..u5F! $$((. >[$w? ? rc|jd\}}}|||df}|j|j|}||dj||||f|jd<y)zShift a token. (Internal)r,N)r$rrappend) r r r r:r r4r5rr(s rr0z Parser.shiftsb::b>UD.,,t||W5   HOOG $x. 2rc|jd\}}}|d|gf}|||f|jd<|jj|d|fy)zPush a nonterminal. (Internal)r,Nr!)r$rH) r r newdfar:r r4r5rr(s rr2z Parser.pushsQ::b>UDw+x. 2 61g./rc*|jj\}}}|j|j|}|W|jr(|jd\}}}|dj |y||_|j |j _yy)zPop a nonterminal. (Internal)Nr,)r$r1rrrHr%r')r popdfapopstatepopnoder(r4r5rs rr1z Parser.popsz$(JJNN$4!',,t||W5  zz#'::b> UDR( ' +/?? ( rr) rrrrrr*r@r.r0r2r1rrrrrs-:?@ 0.H` /0 ;rrN)rrrrobjectrrrrrQs+ K Kn;Vn;r__pycache__/driver.cpython-312.pyc000064400000017642152527262410012772 0ustar00 {|jQdZdZddgZddlZddlZddlZddlZddlZddlm Z m Z m Z m Z m Z GddeZd Z dd Zd Zd Zd Zedk(rej,ee yy)zZParser driver. This provides a high-level interface to parse a file into a syntax tree. z#Guido van Rossum Driver load_grammarN)grammarparsetokentokenizepgenc<eZdZddZd dZd dZd dZd dZd dZy) rNcZ||_|tj}||_||_y)N)rlogging getLoggerloggerconvert)selfrrrs -/usr/lib64/python3.12/lib2to3/pgen2/driver.py__init__zDriver.__init__s* >&&(F  cftj|j|j}|j d}d}dx}x}x}x} } d} |D]H} | \}}}} } |||fk7r:||f|ks J||f|f|\} }|| kr| d| |z zz } | }d}||kr | | ||z } |}|t j t jfvr#| |z } | \}}|jdr|dz }d}|tjk(rtj|}|r/|jjdtj||| |j||| |fr*|r|jjd|j"Sd} | \}}|jdsB|dz }d}Ktj d||| |f) z4Parse a series of tokens and return the syntax tree.rrN z%s %r (prefix=%r)zStop.zincomplete input)rParserrrsetupr COMMENTNLendswithrOPopmaprdebugtok_nameaddtoken ParseErrorrootnode)rtokensrplinenocolumntypevaluestartend line_textprefix quintuples_linenos_columns r parse_tokenszDriver.parse_tokens&s LLt|| 4  1555u5u5sYI1: .D%Y(('50KFF3CU2KK0%*"(H$dh&788F%FFH$ix88F%F(((++66%!$>>$'aKFFuxx}}U+ !!"5"'.."6vGzz$7KK%%g.zzF NFF~~d#! A F""#5#'A Arcdtj|j}|j||Sz*Parse a stream and return the syntax tree.)r generate_tokensreadliner1)rstreamrr$s rparse_stream_rawzDriver.parse_stream_rawVs)))&//:  //rc&|j||Sr3)r7)rr6rs r parse_streamzDriver.parse_stream[s$$VU33rctj|d|5}|j||cdddS#1swYyxYw)z(Parse a file and return the syntax tree.r)encodingN)ioopenr9)rfilenamer<rr6s r parse_filezDriver.parse_file_s0 WWXsX 6&$$VU37 6 6s5>ctjtj|j}|j ||S)z*Parse a string and return the syntax tree.)r r4r=StringIOr5r1)rtextrr$s r parse_stringzDriver.parse_stringds4))"++d*;*D*DE  //r)NN)F)NF) __name__ __module__ __qualname__rr1r7r9r@rDrrrrs!.`0 44 0rctjj|\}}|dk(rd}||zdjt t t jzdzS)Nz.txtr.z.pickle)ospathsplitextjoinmapstrsys version_info)gtheadtails r_generate_pickle_namerVjsP!!"%JD$ v~ $;#c3+;+;"<= = IIrc|tj}| t|n|}|s t||sQ|j d|t j |}|r&|j d| |j||S|Stj}|j||S#t$r}|j d|Yd}~|Sd}~wwxYw)z'Load the grammar (maybe from a pickle).Nz!Generating grammar tables from %szWriting grammar tables to %szWriting failed: %s) r rrV_newerinfor generate_grammardumpOSErrorrGrammarload)rSgpsaveforcerges rrrqs~""$&(j r "bB F2rN 7<  ! !" %  KK6 ; 5r H1H OO  r H  5 0!44 H  5s0B,, C5C  Cctjj|sytjj|sytjj|tjj|k\S)z0Inquire whether file a was written since file b.FT)rKrLexistsgetmtime)abs rrXrXsQ 77>>!  77>>!  77  A "''"2"21"5 55rc tjj|r t|St tjj |}t j||}tj}|j||S)aNormally, loads a pickled grammar by doing pkgutil.get_data(package, pickled_grammar) where *pickled_grammar* is computed from *grammar_source* by adding the Python version and using a ``.pickle`` extension. However, if *grammar_source* is an extant file, load_grammar(grammar_source) is called instead. This facilitates using a packaged grammar file when needed but preserves load_grammar's automatic regeneration behavior when possible. ) rKrLisfilerrVbasenamepkgutilget_datarr]loads)packagegrammar_source pickled_namedatarbs rload_packaged_grammarrssf ww~~n%N++()9)9.)IJL   G\ 2DAGGDM Hrc|stjdd}tjtjtj d|D]}t |ddy)zMain program, when run as a script: produce grammar pickle files. Calls load_grammar for each argument, a path to a grammar text file. rNz %(message)s)levelr6formatT)r`ra)rQargvr basicConfigINFOstdoutr)argsrSs rmainr|sL xx| gll3::,.Rd$/ r__main__)z Grammar.txtNTFN)__doc__ __author____all__r=rKr rlrQrrrrr r objectrrVrrXrsr|rEexitintrHrrrs 3 ^ $  43J0VJ0ZJ'+04 *6 (  z CHHSTV_r__pycache__/__init__.cpython-312.opt-1.pyc000064400000000263152527262410014164 0ustar00 {|jdZy)zThe pgen2 package.N)__doc__//usr/lib64/python3.12/lib2to3/pgen2/__init__.pyrs r__pycache__/__init__.cpython-312.opt-2.pyc000064400000000225152527262410014163 0ustar00 {|jy)Nr//usr/lib64/python3.12/lib2to3/pgen2/__init__.pyrs r__pycache__/tokenize.cpython-312.pyc000064400000050711152527262410013321 0ustar00 {|jR dZdZdZddlZddlZddlmZmZddlddl m Z e e Dcgc] }|dd k7s |c}gd zZ [ e d Zd Zd ZdZdZdZeedezzeezZdZdZdZdZeddZeeeeeZdZeddeezZdezZeeeZ ede dzZ!ee!e eZ"dZ#dZ$d Z%d!Z&d"Z'ee'd#ze'd$zZ(ee'd%ze'd&zZ)ed'd(d)d*d+d,d-d.d/ Z*d0Z+ed1d2d3Z,ee*e+e,Z-ee"e-e)eZ.ee.zZ/ee'd4zed5dze'd6zed7dzZ0edee(Z1eee1e"e-e0ezZ2e3ejhe/e2e%e&f\Z5Z6Z7Z8ed8d9d:d;ed8d9dzZ9ejhe#ejhe$e7e8d?e9Dcic]}|d#e7 c}e9Dcic]}|d$e8 c}e9Dcic]}|dc}Z:d#d$he9Dchc]}|d# c}ze9Dchc]}|d$ c}zZ;d5d7he9Dchc]}|d5 c}ze9Dchc]}|d7 c}zZZ?GdCdDe>Z@dEZAeAfdFZBdGZCGdHdIZDejhdJejZFejhdKejZGdLZHdMZIdNZJdOZKeLdPk(r\ddlMZMeNeMjdkDr&eBePeMjdjyeBeMjjyycc}w#e$reZ YwxYwcc}wcc}wcc}wcc}wcc}wcc}wcc}w)QaTokenization help for Python programs. generate_tokens(readline) is a generator that breaks a stream of text into Python tokens. It accepts a readline-like method which is called repeatedly to get the next line of input (or "" for EOF). It generates 5-tuples with these members: the token type (see token.py) the token (a string) the starting (row, column) indices of the token (a 2-tuple of ints) the ending (row, column) indices of the token (a 2-tuple of ints) the original line (string) It is designed to match the working of the Python tokenizer exactly, except that it produces COMMENT tokens for comments and gives type OP for all operators Older entry points tokenize_loop(readline, tokeneater) tokenize(readline, tokeneater=printtoken) are the same, except instead of generating tokens, tokeneater is a callback function to which the 5 fields described above are passed as 5 arguments, each time a new token is found.zKa-Ping Yee z@GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip MontanaroN)BOM_UTF8lookup)*)token_)tokenizegenerate_tokens untokenizec0ddj|zdzS)N(|))joinchoicess //usr/lib64/python3.12/lib2to3/pgen2/tokenize.pygroupr0sC#((7"33c99ct|dzS)Nrrrs ranyr1s%/C//rct|dzS)N?rrs rmayber2sE7Oc11rc,tfdDS)Nc3K|]5}dzD]+}|j|jk7s%||z-7yw))N)casefold).0xyls r z _combinations..4s8!!e)Qqzz|qzz|/KA)qs.> >)set)r#s`r _combinationsr&3s  rz[ \f\t]*z #[^\r\n]*z\\\r?\nz\w+z0[bB]_?[01]+(?:_[01]+)*z(0[xX]_?[\da-fA-F]+(?:_[\da-fA-F]+)*[lL]?z0[oO]?_?[0-7]+(?:_[0-7]+)*[lL]?z[1-9]\d*(?:_\d+)*[lL]?z0[lL]?z[eE][-+]?\d+(?:_\d+)*z\d+(?:_\d+)*\.(?:\d+(?:_\d+)*)?z\.\d+(?:_\d+)*z \d+(?:_\d+)*z\d+(?:_\d+)*[jJ]z[jJ]z[^'\\]*(?:\\.[^'\\]*)*'z[^"\\]*(?:\\.[^"\\]*)*"z%[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''z%[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""z'(?:[uUrRbBfF]|[rR][fFbB]|[fFbBuU][rR])?'''"""z'[^\n'\\]*(?:\\.[^\n'\\]*)*'z"[^\n"\\]*(?:\\.[^\n"\\]*)*"z\*\*=?z>>=?z<<=?z<>z!=z//=?z->z[+\-*/%&@|^=<>]=?~z[][(){}]z\r?\nz:=z[:;.,`@]z'[^\n'\\]*(?:\\.[^\n'\\]*)*'z"[^\n"\\]*(?:\\.[^\n"\\]*)*"rRfFbB>UuURUruRur)r*r+r'r(c eZdZy) TokenErrorN__name__ __module__ __qualname__rrr:r:rr:c eZdZy)StopTokenizingNr;r?rrrBrBr@rrBc `|\}}|\}}td||||t|t|fzy)Nz%d,%d-%d,%d: %s %s)printtok_namerepr) typerxxx_todo_changemexxx_todo_changeme1linesrowscolerowecols r printtokenrOs<$LT4%LT4 tT4$e= >?rc< t||y#t$rYywxYw)a: The tokenize() function accepts two parameters: one representing the input stream, and one providing an output mechanism for tokenize(). The first parameter, readline, must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string. The second parameter, tokeneater, must also be a callable object. It is called once for each token, with five arguments, corresponding to the tuples generated by generate_tokens(). N) tokenize_looprB)readline tokeneaters rr r s# h +    s  c.t|D]}|| yN)r )rRrS token_infos rrQrQs%h/ J0rc$eZdZdZdZdZdZy) Untokenizerc.g|_d|_d|_y)Nrr)tokensprev_rowprev_col)selfs r__init__zUntokenizer.__init__s   rc|\}}||jksJ||jz }|r|jjd|zyy)N )r[r\rZappend)r]startrowcol col_offsets radd_whitespacezUntokenizer.add_whitespacesJSdmm###4==(  KK  sZ/ 0 rcf|D]}t|dk(r|j||np|\}}}}}|j||jj ||\|_|_|ttfvsw|xj dz c_d|_dj|jS)Nrrr) lencompatrfrZrar[r\NEWLINENLr)r]iterablettok_typerrbendrJs rr zUntokenizer.untokenizesA1v{ Ax(01 -HeUC    & KK  u %+. (DM4=GR=( " ! wwt{{##rcd}g}|jj}|\}}|ttfvr|dz }|tt fvrd}|D]}|dd\}}|ttt tfvr|dz }|tk(r|j|C|tk(r|j]|tt fvrd}n|r|r ||dd}||y)NFr`Trh) rZraNAMENUMBERrkrlASYNCAWAITINDENTDEDENTpop) r]rrm startlineindents toks_appendtoknumtokvaltoks rrjzUntokenizer.compats kk((  dF^ # cMF gr] "IC !WNFF$u55# v&6! GR=( wGBK(!  #rN)r<r=r>r^rfr rjr?rrrXrXs 1 $ rrXz&^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)s^[ \t\f]*(?:[#\r\n]|$)c|ddjjdd}|dk(s|jdry|dvs|jdry |S) z(Imitates get_normal_name in tokenizer.c.N r-utf-8zutf-8-)zlatin-1 iso-8859-1z iso-latin-1)zlatin-1-z iso-8859-1-z iso-latin-1-r)lowerreplace startswith)orig_encencs r_get_normal_namersX 3B-    ' 'S 1C g~1 66 ~~AB Orcdd}d}fd}fd}|}|jtr d|dd}d}|s|gfS||}|r||gfStj|s||gfS|}|s||gfS||}|r|||gfS|||gfS) a The detect_encoding() function is used to detect the encoding that should be used to decode a Python source file. It requires one argument, readline, in the same way as the tokenize() generator. It will call readline a maximum of twice, and return the encoding used (as a string) and a list of any lines (left as bytes) it has read in. It detects the encoding from the presence of a utf-8 bom or an encoding cookie as specified in pep-0263. If both a bom and a cookie are present, but disagree, a SyntaxError will be raised. If the encoding cookie is an invalid charset, raise a SyntaxError. Note that if a utf-8 bom is found, 'utf-8-sig' is returned. If no encoding is specified, then the default of 'utf-8' will be returned. FNrcF S#t$r tcYSwxYwrU) StopIterationbytes)rRsr read_or_stopz%detect_encoding..read_or_stops& :  7N s    c> |jd}tj|}|syt |j d} t |}r|jdk7r td|dz }|S#t$rYywxYw#t$rtd|zwxYw)Nasciirzunknown encoding: rzencoding problem: utf-8z-sig) decodeUnicodeDecodeError cookie_rematchrrr LookupError SyntaxErrorname)rJ line_stringrencodingcodec bom_founds r find_cookiez$detect_encoding..find_cookie s ++g.K ,#EKKN3 ?8$E zzW$!";<<  H#"   ?2X=> > ?sA5 B5 BBBTz utf-8-sig)rrblank_rer)rRrdefaultrrfirstsecondrs` @rdetect_encodingrs$IHG , NE ! ab  {5!H%  >>%  ^F 6"H%(( UFO ##rc8t}|j|S)aTransform tokens back into Python source code. Each element returned by the iterable must be a token sequence with at least two elements, a token number and token value. If only two tokens are passed, the resulting output is poor. Round-trip invariant for full input: Untokenized source will match input source exactly Round-trip invariant for limited input: # Output text will tokenize the back to the input t1 = [tok[:2] for tok in generate_tokens(f.readline)] newcode = untokenize(t1) readline = iter(newcode.splitlines(1)).next t2 = [tok[:2] for tokin generate_tokens(readline)] assert t1 == t2 )rXr )rmuts rr r :s$ B == ""rc# Kdx}x}}d\}}d}dg}d}d} d} d} |} |dz}dt| }} |r| s tdj| }|r4|j dx} }t || d|z||f|| zfd\}}d}n|r0| ddd k7r(| d dd k7r t || z|t| f|fd}d}|| z}|| z}|dk(r|s| sn/d}| |krA| | d k(r|dz}n(| | d k(r|tzdztz}n | | dk(rd}nn | dz} | |krA| |k(rn|r|d}| | dvr| | dk(r]| | djd}| t|z}t||| f|| t|zf| ft| |d||f|t| f| fn,ttf| | dk(| | d|| f|t| f| f||dkDr%|j|t| d| |df|| f| f||dkrC||vrtdd|| | f|dd}| r| |dk\rd} d} d} td|| f|| f| f||dkrC| r#| r!| |dk\rd} d} d} n| std|dfd}| |krtj| | }|r]|j!d\}}||f||f|} }}| ||| |}}|t"j$vs |dk(r|dk7rt&|||| fn&|dvr)t(}|dkDrt}n| rd} |r|d}||||| fn|dk(r+|j*drJ|r|d}t|||| fn|t,vrYt.|}|j| | }|r/|j d} | || }|r|d}t |||| f| fnu||f}| |d}| }no|t0vs|ddt0vs |ddt0vrR|ddk(r4||f}t.|xst.|dxs t.|d}| |dd}}| }n|r|d}t |||| fn|j3r|dvr| r|dk(rt4nt6|||| ft8|||| f}|dk(r|s|}|dvr=|r;|dt8k(r/|ddk(r'|dk(rd} |d} t4|d|d|d|dfd}|r|d}|nd|d k(r|r|d}t|||| f| fd}nE|d!vr|dz}n |d"vr|dz }|r|d}t:|||| fnt | | || f|| dzf| f| dz} | |kr|r|d}|ddD]}td|df|dfdft<d|df|dfdfy#t$rd} Y%wxYww)#a4 The generate_tokens() generator requires one argument, readline, which must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string. Alternately, readline can be a callable function terminating with StopIteration: readline = open(myfile).next # Example of alternate readline The generator produces 5-tuples with these members: the token type; the token string; a 2-tuple (srow, scol) of ints specifying the row and column where the token begins in the source; a 2-tuple (erow, ecol) of ints specifying the row and column where the token ends in the source; and the line on which the token was found. The line passed is the physical line. r)rrNFrrzEOF in multi-line stringz\ z\ r`  z# #z rrz3unindent does not match any outer indentation levelz zEOF in multi-line statement.T rhr)asyncawaitr)defforr\z([{z)]})rrir:rrpSTRING ERRORTOKENtabsizerstripCOMMENTrlrarwIndentationErrorrx pseudoprogspanstringdigitsrtrkendswith triple_quotedendprogs single_quoted isidentifierrurvrsOP ENDMARKER)rRlnumparenlev continuedcontstrneedcontcontliner{stashed async_defasync_def_indent async_def_nlrJposmaxstrstartendprogendmatchrpcolumn comment_tokennl_pos pseudomatchrbsposeposrinitialnewlinerindents rr r Os #$#D#8iGXHcGGIL  :Daxc$iS  !;XFF}}T*H$LLO+cwds3$ho??$)!d23i61d23i86K!7T>#dCI%6BB!D.#d? ]9F)9#fqjV#Y$&&'/A2Ew1N#Y$&Ag ) cz5 CyG#9#$(J$5$5f$=M 3}#55F"M #;sS5G/G(H$PPtFG} &>D#d)+'"+.M! $ #$  !>q JJICi$**45K(--a0 s#'-$cCd!%eC$u+wfmm+sNu|!5$d;;&%G!|"$"'+ % "&"E4t<<^-u~~d333% "&"E4t<<m+&uoG&}}T37H&ll1o $U3"")M&*G%udT3KFF$(%="&uv,#' -"1I."1I.RyD($(%=#+G#4$6q8J$6#+E!H#5 ,0L!#'"")M&*G%udD$??))+ 22$,1W,<5%#($d#<<$dD9C'"% .#$+AJ$$6$+AJ'$9$~,0 3:2; 0#('!*#*1:wqz#*1:#//'+G% "&I_% "&udT3K>> !I%'HqL E)hl8% "&udD$77!49 #;s1u t==AgSCii ~ !"+rD!9tQi44 b4)dAY 33E D s<U T;C:U D U KU 7U ; U U  U  U __main__)S__doc__ __author__ __credits__rrecodecsrrlib2to3.pgen2.tokenrrdir__all__r NameErrorstrrrrr& WhitespaceCommentIgnoreName Binnumber Hexnumber Octnumber Decnumber IntnumberExponent PointfloatExpfloat Floatnumber ImagnumberNumberSingleDoubleSingle3Double3 _litprefixTripleStringOperatorBracketSpecialFunny PlainTokenTokenContStr PseudoExtras PseudoTokenmapcompile tokenprogr single3prog double3prog _strprefixesrrrr Exceptionr:rBrOr rQrXASCIIrrrrr r r<sysriargvopenrRstdin)r!prefixs00rrs#0* F #!%j 0jAaDCK1j 04, ,  :/1   c*z12 2U7^ C  & 7 . +X 6 )Y 9 = # 57H IERZO [ X %J) & g(= > z; 2 $ # 2 2 7 zE!:#5 6 z;;;; = GWeU%%   % - h) 65&$ /   ;;c:&';;c:&' (Z&1 5vugtLL 25JJ Wg638/ :{K#sC%#sC%&&' F#*"**V*<{ 99EFv&~{*F 9:FFv&~{*F 9+77,vt|,7  9 EN"./,xs^,/0"./,xs^,/0  #J ,- fxq\ -. ,- fxq\ -. !!%Y%? #- & 6 6 p BJJ@"(( K 2::0"(( ; G$R#*`4D z 388}q(4 #4#=#=> 399%% &Y 1  E\GF70/.-sF L$L$L) L7 L<) M> M M * M> M)L43L4__pycache__/driver.cpython-312.opt-2.pyc000064400000015651152527262410013730 0ustar00 {|jQ dZddgZddlZddlZddlZddlZddlZddlmZm Z m Z m Z m Z Gdde ZdZ dd Zd Zd Zd Zed k(rej*ee yy)z#Guido van Rossum Driver load_grammarN)grammarparsetokentokenizepgenc<eZdZddZd dZd dZd dZd dZd dZy) rNcZ||_|tj}||_||_yN)rlogging getLoggerloggerconvert)selfrrrs -/usr/lib64/python3.12/lib2to3/pgen2/driver.py__init__zDriver.__init__s* >&&(F  cD tj|j|j}|j d}d}dx}x}x}x} } d} |D]6} | \}}}} } |||fk7r(|\} }|| kr| d| |z zz } | }d}||kr | | ||z } |}|t j t jfvr#| |z } | \}}|jdr|dz }d}|tjk(rtj|}|r/|jjdtj||| |j||| |fr*|r|jjd|j"Sd} | \}}|jds0|dz }d}9tj d||| |f)Nrr z%s %r (prefix=%r)zStop.zincomplete input)rParserrrsetupr COMMENTNLendswithrOPopmaprdebugtok_nameaddtoken ParseErrorrootnode)rtokensr plinenocolumntypevaluestartend line_textprefix quintuples_linenos_columns r parse_tokenszDriver.parse_tokens&sB LLt|| 4  1555u5u5sYI1: .D%Y((%*"(H$dh&788F%FFH$ix88F%F(((++66%!$>>$'aKFFuxx}}U+ !!"5"'.."6vGzz$7KK%%g.zzF NFF~~d#! A F""#5#'A Arcf tj|j}|j||Sr )r generate_tokensreadliner2)rstreamr r%s rparse_stream_rawzDriver.parse_stream_rawVs,8))&//:  //rc( |j||Sr )r7)rr6r s r parse_streamzDriver.parse_stream[s8$$VU33rc tj|d|5}|j||cdddS#1swYyxYw)Nr)encoding)ioopenr9)rfilenamer<r r6s r parse_filezDriver.parse_file_s36 WWXsX 6&$$VU37 6 6s6?c tjtj|j}|j ||Sr )r r4r=StringIOr5r2)rtextr r%s r parse_stringzDriver.parse_stringds78))"++d*;*D*DE  //r)NN)F)NF) __name__ __module__ __qualname__rr2r7r9r@rDrrrrs!.`0 44 0rctjj|\}}|dk(rd}||zdjt t t jzdzS)Nz.txtr.z.pickle)ospathsplitextjoinmapstrsys version_info)gtheadtails r_generate_pickle_namerVjsP!!"%JD$ v~ $;#c3+;+;"<= = IIrc |tj}| t|n|}|s t||sQ|j d|t j |}|r&|j d| |j||S|Stj}|j||S#t$r}|j d|Yd}~|Sd}~wwxYw)Nz!Generating grammar tables from %szWriting grammar tables to %szWriting failed: %s) rrrV_newerinfor generate_grammardumpOSErrorrGrammarload)rSgpsaveforcerges rrrqs1 ~""$&(j r "bB F2rN 7<  ! !" %  KK6 ; 5r H1H OO  r H  5 0!44 H  5s1B-- C6CCc tjj|sytjj|sytjj|tjj|k\S)NFT)rKrLexistsgetmtime)abs rrXrXsT: 77>>!  77>>!  77  A "''"2"21"5 55rc" tjj|r t|St tjj |}t j||}tj}|j||Sr ) rKrLisfilerrVbasenamepkgutilget_datarr]loads)packagegrammar_source pickled_namedatarbs rload_packaged_grammarrssk  ww~~n%N++()9)9.)IJL   G\ 2DAGGDM Hrc |stjdd}tjtjtj d|D]}t |ddy)Nrz %(message)s)levelr6formatT)r`ra)rQargvr basicConfigINFOstdoutr)argsrSs rmainr|sQ xx| gll3::,.Rd$/ r__main__)z Grammar.txtNTFN) __author____all__r=rKrrlrQrrrrr r objectrrVrrXrsr|rEexitintrHrrrs 3 ^ $  43J0VJ0ZJ'+04 *6 (  z CHHSTV_r__pycache__/literals.cpython-312.opt-1.pyc000064400000004124152527262410014244 0ustar00 {|jc VdZddlZddddddd d d d d ZdZdZdZedk(reyy)zADHI I TE2A q6M  VD! A q6M TADHIt S T  VCdJKQU U Vs" B: B-B*-Cc|d}|dd|dzk(r|dz}|t|t| }tjdt|S)Nrz)\\(\'|\"|\\|[abfnrtv]|x.{0,2}|[0-7]{1,3}))rresubr))sqs r( evalStringr0(sQ !A!u!| aC #a&#a&A 66> JJctdD]7}t|}t|}t|}||k7s*t ||||9y)N)ranger!reprr0print)r'cr.es r(testr92s@ 3Z F G qM 6 !Q1  r1__main__)__doc__r,rr)r0r9__name__r1r(r>sWC  *K zFr1__pycache__/conv.cpython-312.pyc000064400000026757152527262410012453 0ustar00 {|j%JdZddlZddlmZmZGddej Zy)aConvert graminit.[ch] spit out by pgen to Python code. Pgen is the Python parser generator. It is useful to quickly create a parser from a grammar file in Python's grammar notation. But I don't want my parsers to be written in C (yet), so I'm translating the parsing tables to Python data structures and writing a Python parse engine. Note that the token numbers are constants determined by the standard Python tokenizer. The standard token module defines these numbers and their names (the names are not used much). The token numbers are hardcoded into the Python tokenizer and into pgen. A Python implementation of the Python tokenizer is also available, in the standard tokenize module. On the other hand, symbol numbers (representing the grammar's non-terminals) are assigned by pgen based on the actual grammar input. Note: this module is pretty much obsolete; the pgen module generates equivalent grammar tables directly from the Grammar.txt input file without having to invoke the Python pgen C program. N)grammartokenc(eZdZdZdZdZdZdZy) Convertera2Grammar subclass that reads classic pgen output files. The run() method reads the tables as produced by the pgen parser generator, typically contained in two C files, graminit.h and graminit.c. The other methods are for internal use only. See the base class for more documentation. ch|j||j||jy)z_[] = { {, }, ... }; - followed by a state array, of the form: static state states_[] = { {, arcs__}, ... }; rrNFrrz#include "pgenheaders.h" z#include "grammar.h" z static arc z)static arc arcs_(\d+)_(\d+)\[(\d+)\] = {$z\s+{(\d+), (\d+)},$z}; z'static state states_(\d+)\[(\d+)\] = {$z\s+{(\d+), arcs_(\d+)_(\d+)},$zstatic dfa dfas\[(\d+)\] = {$z0\s+{(\d+), "(\w+)", (\d+), (\d+), states_(\d+),$z\s+("(?:\\\d\d\d)*")},$z!static label labels\[(\d+)\] = {$z\s+{(\d+), (0|"\w+")},$0zgrammar _PyParser_Grammar = { z \s+(\d+),$z dfas, z\s+{(\d+), labels},$z \s+(\d+)$)rrrnext startswithrrlistmaprrrangeappendlenstatesgrouprreval enumerateorddfaslabelsstart StopIteration)!r r r!r"r#r$allarcsr6r%nmkarcs_ijststater;ndfasr&r'xyzfirst rawbitsetcbyter<nlabelsr=s! rr zConverter.parse_graminit_cTs8 XA axa33Cfd^C3axa//?&$?/axaoom,//-0XXJ"$)FD>)rs3 451aqA#)!8T!WDF"8$?B-~-2C 56DAqKKA' " &axav~5~5~"&A%axa//-0 DdKB %~ %2C-.DAqF # 3fd^ 3#E1X%axaXX?F)FD>)rs3 451aq!t}CI~5~5~ T" MM% !!8T!WDF6> 1FD> 1>!!8T!WDFCoom,D  XX6 =!FD>!rBHHQK uA!!8T!WDFM B %~ %2XXa[F"3sBHHQ1a,@#ABOFAq!%%f-7 G&$ G7%%f-7 G&$ G76 )FD> )61IEE ? 2VTN 2?!!8T!WDF4d;B %~ %2ERXXa[)I!),11vqAq!t})*acAg"- "5>DL-.axav~-~-~ axa XX:D A!FD>!rbhhqk"wA!!8T!WDF4d;B %~ %299;DAqAACxG MM1a& ! axav~-~-~ axa88H64.H8axa XXmT *!FD>!rBHHQK DII&&&axa{"2VTN2"axa XX-t 4!FD>!rbhhqk"#dkk**:VTN:*axa XXlD )!FD>!rBHHQK ***:VTN:* axav~-~-~ %!!8T!WDF %vtn $1K  37 8 D   s) ]0^0 ^9^^ ^#"^#ci|_i|_t|jD]?\}\}}|tj k(r|||j|<.|1||j|<Ay)z1Create additional useful structures. (Internal).N)keywordstokensr9r<rNAME)r ilabeltypevalues rr zConverter.finish_offs^  %.t{{%; !FMT5uzz!e&7'- e$$* D! &c%J+rr)r]rpgen2rrGrammarrr^rrras&4 !]+]+r__pycache__/pgen.cpython-312.opt-1.pyc000064400000042465152527262410013370 0ustar00 {|j6ddlmZmZmZGddejZGddeZGddeZGdd eZ d d Z y ) )grammartokentokenizec eZdZy) PgenGrammarN)__name__ __module__ __qualname__+/usr/lib64/python3.12/lib2to3/pgen2/pgen.pyrrsr rc|eZdZddZdZdZdZdZdZdZ d Z d Z d Z d Z d ZdZdZdZddZdZdZy)ParserGeneratorNc<d}|t|d}|j}||_||_t j |j |_|j|j\|_ |_ ||i|_ |jy)Nzutf-8)encoding)openclosefilenamestreamrgenerate_tokensreadline generatorgettokenparsedfas startsymbolfirst addfirstsets)selfrr close_streams r __init__zParserGenerator.__init__ s >(W5F!<==KKCIIe$4 56 d#  HHOOF #-3T__Q5M,NAFF1??4( )//$"2"23r cv|j|}i}t|D]}|j||}d||<|SNr)rr-r1)rr7r9rawfirstrr=ilabels r r5zParserGenerator.make_first4sD::d#H%E__Q.FE&M& r ct|j}|djr||jvrX||jvr|j|S|jj |j|df||j|<|St t|d}||jvr|j|S|jj |df||j|<|St|}|djrY||jvr|j|S|jj tj|f||j|<|Stj|}||jvr|j|S|jj |df||j|<|SNr#)r*labelsisalphar+ symbol2labelr0getattrrtokensevalkeywordsNAMEropmap)rr7r=rCitokenvalues r r1zParserGenerator.make_label=sQXX 8   'ANN*>>%00HHOOQ__U%;T$BC,2ANN5)!M!t4QXX%88F++HHOOVTN3'-AHHV$!MKEQx!AJJ&::e,,HHOOUZZ$78(.AJJu%!M!u-QXX%88F++HHOOVTN3'-AHHV$!Mr ct|jj}|j|D]"}||jvs|j |$yN)r%rr&r'r calcfirst)rr8r9s r rzParserGenerator.addfirstsetsksBTYY^^%& D4::%t$r c 8|j|}d|j|<|d}i}i}|jjD]\}}||jvrd||jvr|j|}|.t d|z|j ||j|}|j ||||<xd||<|di||<i} |jD]/\}} | D]%} | | vrt d|d| d|d| | || | <'1||j|<y)Nr#zrecursion for rule %rrzrule z is ambiguous; z is in the first sets of z as well as )rrr.r/ ValueErrorrSupdate) rr9r;r<totalset overlapcheckr=r>fsetinverseitsfirstsymbols r rSzParserGenerator.calcfirstss;iio 4A  ::++-KE4 !DJJ&::e,D|()@4)GHHNN5)::e,D%&* U#"#',aj U#.+113OE8"W$$&*FE76?&LMM#( # 4$ 4r ci}d}|jtjk7r|jtjk(r.|j |jtjk(r.|j tj }|j tjd|j\}}|j tj|j||}t|}|j|t|}|||<||}|jtjk7r||fS)N:) typer ENDMARKERNEWLINErexpectrMOP parse_rhsmake_dfar* simplify_dfa) rrrr9azr;oldlennewlens r rzParserGenerator.parses ii5??*))u}}, ))u}},;;uzz*D KK# &>>#DAq KK &--1%CXF   c "XFDJ"" #ii5??*$[  r c  fd} fd t|||g}|D]}i}|jD]2}|jD]!\}} |  | |j|i#4t |j D]L\}} |D]} | j| k(snt| |} |j | |j| |N|S)Nci}|||SrRr )r<base addclosures r closurez)ParserGenerator.make_dfa..closuresD ud #Kr cZ||vryd||<|jD]\}}| ||yrAr.)r<rmr=r>rns r rnz,ParserGenerator.make_dfa..addclosures7}DK$zz t=tT* *r )DFAStatenfasetr. setdefaultr-r/r0addarc) rr6finishror4r<r.nfastater=r>rsstrns @r rezParserGenerator.make_dfas  +75>623ED!LL#+==KE4("4)CD$1)"( !5 v ByyF*!"&&1BMM"% R'"6  r cDtd||g}t|D]\}}td|||uxrdxsd|jD]X\}}||vr|j|} nt |} |j ||td| zItd|| fzZy)NzDump of NFA for State(final)z -> %d %s -> %d)print enumerater.r2r*r0) rr9r6rvtodor:r<r=r>js r dump_nfazParserGenerator.dump_nfas &w!$HAu )Q =I C D$zz t4< 4(AD AKK%=+/*.E1:56 *(r c td|t|D]n\}}td||jxrdxsdt|jj D]$\}}td||j |fz&py)NzDump of DFA forrzr{r|r})r~rr3r-r.r/r2)rr9r;r:r<r=r>s r dump_dfazParserGenerator.dump_dfass &!#HAu )Q ;) Ar B%ejj&6&6&89 tnsyy'??@ :'r cd}|r`d}t|D]L\}}t|dzt|D],}||}||k(s||=|D]}|j||d}LN|r_yy)NTFr)rranger* unifystate)rr;changesr:state_irstate_jr<s r rfzParserGenerator.simplify_dfas{G'n 7qsCH-A!!fG')F%(E!,,Wg>&)"&.-r c|j\}}|jdk7r||fSt}t}|j||j||jdk(rU|j |j\}}|j||j||jdk(rU||fS)N|) parse_altrPNFAStaterur)rrgrhaazzs r rdzParserGenerator.parse_rhss~~1 :: a4KBB IIaL HHRL**# ~~'1 !  **# r6Mr ch|j\}}|jdvs,|jtjtj fvrb|j\}}|j ||}|jdvr5|jtjtj fvrb||fS)N)([) parse_itemrPr_rrMSTRINGru)rrgbr7ds r rzParserGenerator.parse_alt s 1zzZ'yyUZZ66??$DAq HHQKA zzZ'yyUZZ66!t r cz|jdk(rX|j|j\}}|jtj d|j |||fS|j\}}|j}|dvr||fS|j|j ||dk(r||fS||fS)Nr])+*r)rPrrdrbrrcru parse_atom)rrgrhrPs r rzParserGenerator.parse_items ::  MMO>>#DAq KK# & HHQKa4K??$DAqJJEJ&!t MMO HHQK|!t !t r c|jdk(rG|j|j\}}|jtj d||fS|j tjtjfvrDt}t}|j||j|j||fS|jd|j |jy)Nr)z+expected (...) or NAME or STRING, got %s/%s) rPrrdrbrrcr_rMrrru raise_error)rrgrhs r rzParserGenerator.parse_atom(s ::  MMO>>#DAq KK# &a4K YY5::u||4 4 A A HHQ # MMOa4K   J!YY  4r c|j|k7s|8|j|k7r)|jd|||j|j|j}|j|S)Nzexpected %s/%s, got %s/%s)r_rPrr)rr_rPs r rbzParserGenerator.expect9sX 99 !2tzzU7J   8!5$))TZZ A   r c0t|j}|dtjtjfvr;t|j}|dtjtjfvr;|\|_|_|_|_|_ yrE) r>rrCOMMENTNLr_rPbeginendline)rtups r rzParserGenerator.gettokenAsp4>>"!f))8;;77t~~&C!f))8;;77AD> 4:tz48TYr c |r ||z}t ||j |j d|j d|jf#dj|gttt|z}YnxYw)N r#r)joinr%mapstr SyntaxErrorrrr)rmsgargss r rzParserGenerator.raise_errorHsq  =Dj# txx{ $ TYY 89 9 =hhutCTN';;<s A.A7rR)rr r r!r?r5r1rrSrrerrrfrdrrrrbrrr r r rr s` 2,"\%$<!0"H7 A*"(4"E9r rceZdZdZddZy)rcg|_yrRrq)rs r r!zNFAState.__init__Ss  r Nc>|jj||fyrR)r.r0rr>r=s r ruzNFAState.addarcVs %'r rR)rr r r!rur r r rrQs (r rc(eZdZdZdZdZdZdZy)rrc2||_||v|_i|_yrR)rsr3r.)rrsfinals r r!zDFAState.__init__]s   r c"||j|<yrRrqrs r ruzDFAState.addarces  %r cp|jjD]\}}||us ||j|<yrR)r.r/)roldnewr=r>s r rzDFAState.unifystateks099??,KE4s{#& % -r c|j|jk7ryt|jt|jk7ry|jjD]$\}}||jj |us$yy)NFT)r3r*r.r/get)rotherr=r>s r __eq__zDFAState.__eq__psj <<5== ( tyy>S_ ,99??,KE45::>>%00-r N)rr r r!rurr__hash__r r r rrrr[s ' Hr rrc8t|}|jSrR)rr?)rps r generate_grammarrs!A >> r N)z Grammar.txt) r|rrrGrammarrobjectrrrrrr r r rsJ '& '// E9fE9N (v(#v#Jr __pycache__/literals.cpython-312.pyc000064400000005041152527262410013304 0ustar00 {|jc VdZddlZddddddd d d d d ZdZdZdZedk(reyy)z>$     T "C   sQR u:>ADHI I TE2A q6M  VD! A q6M TADHIt S T  VCdJKQU U Vs5 B$ C$B=Cc|jds$|jdsJt|dd|d}|dd|dzk(r|dz}|j|sJt|t| dt|dt|zk\sJ|t|t| }t j dt |S)Nr r rrrz)\\(\'|\"|\\|[abfnrtv]|x.{0,2}|[0-7]{1,3}))rreprendswithrresubr))sqs r( evalStringr2(s <<  S 1>4"1;> 1 !A!u!| aC ::a=+$q#a&{++= q6Qs1vX   #a&#a&A 66> JJctdD]7}t|}t|}t|}||k7s*t ||||9y)N)ranger!r,r2print)r'cr0es r(testr:2s@ 3Z F G qM 6 !Q1  r3__main__)__doc__r.rr)r2r:__name__r3r(r?sWC  *K zFr3__pycache__/pgen.cpython-312.opt-2.pyc000064400000042465152527262410013371 0ustar00 {|j6ddlmZmZmZGddejZGddeZGddeZGdd eZ d d Z y ) )grammartokentokenizec eZdZy) PgenGrammarN)__name__ __module__ __qualname__+/usr/lib64/python3.12/lib2to3/pgen2/pgen.pyrrsr rc|eZdZddZdZdZdZdZdZdZ d Z d Z d Z d Z d ZdZdZdZddZdZdZy)ParserGeneratorNc<d}|t|d}|j}||_||_t j |j |_|j|j\|_ |_ ||i|_ |jy)Nzutf-8)encoding)openclosefilenamestreamrgenerate_tokensreadline generatorgettokenparsedfas startsymbolfirst addfirstsets)selfrr close_streams r __init__zParserGenerator.__init__ s >(W5F!<==KKCIIe$4 56 d#  HHOOF #-3T__Q5M,NAFF1??4( )//$"2"23r cv|j|}i}t|D]}|j||}d||<|SNr)rr-r1)rr7r9rawfirstrr=ilabels r r5zParserGenerator.make_first4sD::d#H%E__Q.FE&M& r ct|j}|djr||jvrX||jvr|j|S|jj |j|df||j|<|St t|d}||jvr|j|S|jj |df||j|<|St|}|djrY||jvr|j|S|jj tj|f||j|<|Stj|}||jvr|j|S|jj |df||j|<|SNr#)r*labelsisalphar+ symbol2labelr0getattrrtokensevalkeywordsNAMEropmap)rr7r=rCitokenvalues r r1zParserGenerator.make_label=sQXX 8   'ANN*>>%00HHOOQ__U%;T$BC,2ANN5)!M!t4QXX%88F++HHOOVTN3'-AHHV$!MKEQx!AJJ&::e,,HHOOUZZ$78(.AJJu%!M!u-QXX%88F++HHOOVTN3'-AHHV$!Mr ct|jj}|j|D]"}||jvs|j |$yN)r%rr&r'r calcfirst)rr8r9s r rzParserGenerator.addfirstsetsksBTYY^^%& D4::%t$r c 8|j|}d|j|<|d}i}i}|jjD]\}}||jvrd||jvr|j|}|.t d|z|j ||j|}|j ||||<xd||<|di||<i} |jD]/\}} | D]%} | | vrt d|d| d|d| | || | <'1||j|<y)Nr#zrecursion for rule %rrzrule z is ambiguous; z is in the first sets of z as well as )rrr.r/ ValueErrorrSupdate) rr9r;r<totalset overlapcheckr=r>fsetinverseitsfirstsymbols r rSzParserGenerator.calcfirstss;iio 4A  ::++-KE4 !DJJ&::e,D|()@4)GHHNN5)::e,D%&* U#"#',aj U#.+113OE8"W$$&*FE76?&LMM#( # 4$ 4r ci}d}|jtjk7r|jtjk(r.|j |jtjk(r.|j tj }|j tjd|j\}}|j tj|j||}t|}|j|t|}|||<||}|jtjk7r||fS)N:) typer ENDMARKERNEWLINErexpectrMOP parse_rhsmake_dfar* simplify_dfa) rrrr9azr;oldlennewlens r rzParserGenerator.parses ii5??*))u}}, ))u}},;;uzz*D KK# &>>#DAq KK &--1%CXF   c "XFDJ"" #ii5??*$[  r c  fd} fd t|||g}|D]}i}|jD]2}|jD]!\}} |  | |j|i#4t |j D]L\}} |D]} | j| k(snt| |} |j | |j| |N|S)Nci}|||SrRr )r<base addclosures r closurez)ParserGenerator.make_dfa..closuresD ud #Kr cZ||vryd||<|jD]\}}| ||yrAr.)r<rmr=r>rns r rnz,ParserGenerator.make_dfa..addclosures7}DK$zz t=tT* *r )DFAStatenfasetr. setdefaultr-r/r0addarc) rr6finishror4r<r.nfastater=r>rsstrns @r rezParserGenerator.make_dfas  +75>623ED!LL#+==KE4("4)CD$1)"( !5 v ByyF*!"&&1BMM"% R'"6  r cDtd||g}t|D]\}}td|||uxrdxsd|jD]X\}}||vr|j|} nt |} |j ||td| zItd|| fzZy)NzDump of NFA for State(final)z -> %d %s -> %d)print enumerater.r2r*r0) rr9r6rvtodor:r<r=r>js r dump_nfazParserGenerator.dump_nfas &w!$HAu )Q =I C D$zz t4< 4(AD AKK%=+/*.E1:56 *(r c td|t|D]n\}}td||jxrdxsdt|jj D]$\}}td||j |fz&py)NzDump of DFA forrzr{r|r})r~rr3r-r.r/r2)rr9r;r:r<r=r>s r dump_dfazParserGenerator.dump_dfass &!#HAu )Q ;) Ar B%ejj&6&6&89 tnsyy'??@ :'r cd}|r`d}t|D]L\}}t|dzt|D],}||}||k(s||=|D]}|j||d}LN|r_yy)NTFr)rranger* unifystate)rr;changesr:state_irstate_jr<s r rfzParserGenerator.simplify_dfas{G'n 7qsCH-A!!fG')F%(E!,,Wg>&)"&.-r c|j\}}|jdk7r||fSt}t}|j||j||jdk(rU|j |j\}}|j||j||jdk(rU||fS)N|) parse_altrPNFAStaterur)rrgrhaazzs r rdzParserGenerator.parse_rhss~~1 :: a4KBB IIaL HHRL**# ~~'1 !  **# r6Mr ch|j\}}|jdvs,|jtjtj fvrb|j\}}|j ||}|jdvr5|jtjtj fvrb||fS)N)([) parse_itemrPr_rrMSTRINGru)rrgbr7ds r rzParserGenerator.parse_alt s 1zzZ'yyUZZ66??$DAq HHQKA zzZ'yyUZZ66!t r cz|jdk(rX|j|j\}}|jtj d|j |||fS|j\}}|j}|dvr||fS|j|j ||dk(r||fS||fS)Nr])+*r)rPrrdrbrrcru parse_atom)rrgrhrPs r rzParserGenerator.parse_items ::  MMO>>#DAq KK# & HHQKa4K??$DAqJJEJ&!t MMO HHQK|!t !t r c|jdk(rG|j|j\}}|jtj d||fS|j tjtjfvrDt}t}|j||j|j||fS|jd|j |jy)Nr)z+expected (...) or NAME or STRING, got %s/%s) rPrrdrbrrcr_rMrrru raise_error)rrgrhs r rzParserGenerator.parse_atom(s ::  MMO>>#DAq KK# &a4K YY5::u||4 4 A A HHQ # MMOa4K   J!YY  4r c|j|k7s|8|j|k7r)|jd|||j|j|j}|j|S)Nzexpected %s/%s, got %s/%s)r_rPrr)rr_rPs r rbzParserGenerator.expect9sX 99 !2tzzU7J   8!5$))TZZ A   r c0t|j}|dtjtjfvr;t|j}|dtjtjfvr;|\|_|_|_|_|_ yrE) r>rrCOMMENTNLr_rPbeginendline)rtups r rzParserGenerator.gettokenAsp4>>"!f))8;;77t~~&C!f))8;;77AD> 4:tz48TYr c |r ||z}t ||j |j d|j d|jf#dj|gttt|z}YnxYw)N r#r)joinr%mapstr SyntaxErrorrrr)rmsgargss r rzParserGenerator.raise_errorHsq  =Dj# txx{ $ TYY 89 9 =hhutCTN';;<s A.A7rR)rr r r!r?r5r1rrSrrerrrfrdrrrrbrrr r r rr s` 2,"\%$<!0"H7 A*"(4"E9r rceZdZdZddZy)rcg|_yrRrq)rs r r!zNFAState.__init__Ss  r Nc>|jj||fyrR)r.r0rr>r=s r ruzNFAState.addarcVs %'r rR)rr r r!rur r r rrQs (r rc(eZdZdZdZdZdZdZy)rrc2||_||v|_i|_yrR)rsr3r.)rrsfinals r r!zDFAState.__init__]s   r c"||j|<yrRrqrs r ruzDFAState.addarces  %r cp|jjD]\}}||us ||j|<yrR)r.r/)roldnewr=r>s r rzDFAState.unifystateks099??,KE4s{#& % -r c|j|jk7ryt|jt|jk7ry|jjD]$\}}||jj |us$yy)NFT)r3r*r.r/get)rotherr=r>s r __eq__zDFAState.__eq__psj <<5== ( tyy>S_ ,99??,KE45::>>%00-r N)rr r r!rurr__hash__r r r rrrr[s ' Hr rrc8t|}|jSrR)rr?)rps r generate_grammarrs!A >> r N)z Grammar.txt) r|rrrGrammarrobjectrrrrrr r r rsJ '& '// E9fE9N (v(#v#Jr __pycache__/grammar.cpython-312.opt-2.pyc000064400000007460152527262410014062 0ustar00 {|j ddlZddlmZGddeZdZiZejD]$Zesej\Z Z e ee ee <&[[ [ y)N)tokenc2eZdZ dZdZdZdZdZdZy)Grammarci|_i|_g|_i|_dg|_i|_i|_i|_d|_y)N)rEMPTY) symbol2number number2symbolstatesdfaslabelskeywordstokens symbol2labelstart)selfs ./usr/lib64/python3.12/lib2to3/pgen2/grammar.py__init__zGrammar.__init__LsF  #n    c t|d5}tj|j|tjdddy#1swYyxYw)Nwb)openpickledump__dict__HIGHEST_PROTOCOL)rfilenamefs rrz Grammar.dumpWs77 (D !Q KK q&*A*A B" ! !s 0AAc t|d5}tj|}ddd|jj y#1swY%xYw)Nrb)rrloadrupdate)rrrds rr"z Grammar.load\s>9 (D !Q AA" Q" !s AAcb |jjtj|y)N)rr#rloads)rpkls rr&z Grammar.loadsbs A V\\#./rc  |j}dD]'}t||t||j)|jdd|_|j dd|_|j |_|S)N)r r r rrr) __class__setattrgetattrcopyrr r)rnew dict_attrs rr,z Grammar.copyfsm nn4I CGD)$<$A$A$C D4[[^ [[^ JJ  rc` ddlm}td||jtd||jtd||jtd||j td||j td|jy) Nr)pprints2nn2sr r rr)r0printr r r r rr)rr0s rreportzGrammar.reportssyH! e t!!" e t!!" ht{{ f tyy ht{{ gtzz"rN) __name__ __module__ __qualname__rrr"r&r,r4rrrrs'3j C  0  #rra ( LPAR ) RPAR [ LSQB ] RSQB : COLON , COMMA ; SEMI + PLUS - MINUS * STAR / SLASH | VBAR & AMPER < LESS > GREATER = EQUAL . DOT % PERCENT ` BACKQUOTE { LBRACE } RBRACE @ AT @= ATEQUAL == EQEQUAL != NOTEQUAL <> NOTEQUAL <= LESSEQUAL >= GREATEREQUAL ~ TILDE ^ CIRCUMFLEX << LEFTSHIFT >> RIGHTSHIFT ** DOUBLESTAR += PLUSEQUAL -= MINEQUAL *= STAREQUAL /= SLASHEQUAL %= PERCENTEQUAL &= AMPEREQUAL |= VBAREQUAL ^= CIRCUMFLEXEQUAL <<= LEFTSHIFTEQUAL >>= RIGHTSHIFTEQUAL **= DOUBLESTAREQUAL // DOUBLESLASH //= DOUBLESLASHEQUAL -> RARROW := COLONEQUAL ) rrobjectr opmap_rawopmap splitlineslinesplitopnamer+r8rrrBsp j#fj#^1  f   "D ::<DE4(b  # "drparse.py000064400000017565152527262410006254 0ustar00# Copyright 2004-2005 Elemental Security, Inc. All Rights Reserved. # Licensed to PSF under a Contributor Agreement. """Parser engine for the grammar tables generated by pgen. The grammar table must be loaded first. See Parser/parser.c in the Python distribution for additional info on how this parsing engine works. """ # Local imports from . import token class ParseError(Exception): """Exception to signal the parser is stuck.""" def __init__(self, msg, type, value, context): Exception.__init__(self, "%s: type=%r, value=%r, context=%r" % (msg, type, value, context)) self.msg = msg self.type = type self.value = value self.context = context class Parser(object): """Parser engine. The proper usage sequence is: p = Parser(grammar, [converter]) # create instance p.setup([start]) # prepare for parsing : if p.addtoken(...): # parse a token; may raise ParseError break root = p.rootnode # root of abstract syntax tree A Parser instance may be reused by calling setup() repeatedly. A Parser instance contains state pertaining to the current token sequence, and should not be used concurrently by different threads to parse separate token sequences. See driver.py for how to get input tokens by tokenizing a file or string. Parsing is complete when addtoken() returns True; the root of the abstract syntax tree can then be retrieved from the rootnode instance variable. When a syntax error occurs, addtoken() raises the ParseError exception. There is no error recovery; the parser cannot be used after a syntax error was reported (but it can be reinitialized by calling setup()). """ def __init__(self, grammar, convert=None): """Constructor. The grammar argument is a grammar.Grammar instance; see the grammar module for more information. The parser is not ready yet for parsing; you must call the setup() method to get it started. The optional convert argument is a function mapping concrete syntax tree nodes to abstract syntax tree nodes. If not given, no conversion is done and the syntax tree produced is the concrete syntax tree. If given, it must be a function of two arguments, the first being the grammar (a grammar.Grammar instance), and the second being the concrete syntax tree node to be converted. The syntax tree is converted from the bottom up. A concrete syntax tree node is a (type, value, context, nodes) tuple, where type is the node type (a token or symbol number), value is None for symbols and a string for tokens, context is None or an opaque value used for error reporting (typically a (lineno, offset) pair), and nodes is a list of children for symbols, and None for tokens. An abstract syntax tree node may be anything; this is entirely up to the converter function. """ self.grammar = grammar self.convert = convert or (lambda grammar, node: node) def setup(self, start=None): """Prepare for parsing. This *must* be called before starting to parse. The optional argument is an alternative start symbol; it defaults to the grammar's start symbol. You can use a Parser instance to parse any number of programs; each time you call setup() the parser is reset to an initial state determined by the (implicit or explicit) start symbol. """ if start is None: start = self.grammar.start # Each stack entry is a tuple: (dfa, state, node). # A node is a tuple: (type, value, context, children), # where children is a list of nodes or None, and context may be None. newnode = (start, None, None, []) stackentry = (self.grammar.dfas[start], 0, newnode) self.stack = [stackentry] self.rootnode = None self.used_names = set() # Aliased to self.rootnode.used_names in pop() def addtoken(self, type, value, context): """Add a token; return True iff this is the end of the program.""" # Map from token to label ilabel = self.classify(type, value, context) # Loop until the token is shifted; may raise exceptions while True: dfa, state, node = self.stack[-1] states, first = dfa arcs = states[state] # Look for a state with this label for i, newstate in arcs: t, v = self.grammar.labels[i] if ilabel == i: # Look it up in the list of labels assert t < 256 # Shift a token; we're done with it self.shift(type, value, newstate, context) # Pop while we are in an accept-only state state = newstate while states[state] == [(0, state)]: self.pop() if not self.stack: # Done parsing! return True dfa, state, node = self.stack[-1] states, first = dfa # Done with this token return False elif t >= 256: # See if it's a symbol and if we're in its first set itsdfa = self.grammar.dfas[t] itsstates, itsfirst = itsdfa if ilabel in itsfirst: # Push a symbol self.push(t, self.grammar.dfas[t], newstate, context) break # To continue the outer while loop else: if (0, state) in arcs: # An accepting state, pop it and try something else self.pop() if not self.stack: # Done parsing, but another token is input raise ParseError("too much input", type, value, context) else: # No success finding a transition raise ParseError("bad input", type, value, context) def classify(self, type, value, context): """Turn a token into a label. (Internal)""" if type == token.NAME: # Keep a listing of all used names self.used_names.add(value) # Check for reserved words ilabel = self.grammar.keywords.get(value) if ilabel is not None: return ilabel ilabel = self.grammar.tokens.get(type) if ilabel is None: raise ParseError("bad token", type, value, context) return ilabel def shift(self, type, value, newstate, context): """Shift a token. (Internal)""" dfa, state, node = self.stack[-1] newnode = (type, value, context, None) newnode = self.convert(self.grammar, newnode) if newnode is not None: node[-1].append(newnode) self.stack[-1] = (dfa, newstate, node) def push(self, type, newdfa, newstate, context): """Push a nonterminal. (Internal)""" dfa, state, node = self.stack[-1] newnode = (type, None, context, []) self.stack[-1] = (dfa, newstate, node) self.stack.append((newdfa, 0, newnode)) def pop(self): """Pop a nonterminal. (Internal)""" popdfa, popstate, popnode = self.stack.pop() newnode = self.convert(self.grammar, popnode) if newnode is not None: if self.stack: dfa, state, node = self.stack[-1] node[-1].append(newnode) else: self.rootnode = newnode self.rootnode.used_names = self.used_names grammar.py000064400000014675152527262410006567 0ustar00# Copyright 2004-2005 Elemental Security, Inc. All Rights Reserved. # Licensed to PSF under a Contributor Agreement. """This module defines the data structures used to represent a grammar. These are a bit arcane because they are derived from the data structures used by Python's 'pgen' parser generator. There's also a table here mapping operators to their names in the token module; the Python tokenize module reports all operators as the fallback token code OP, but the parser needs the actual token code. """ # Python imports import collections import pickle # Local imports from . import token, tokenize class Grammar(object): """Pgen parsing tables conversion class. Once initialized, this class supplies the grammar tables for the parsing engine implemented by parse.py. The parsing engine accesses the instance variables directly. The class here does not provide initialization of the tables; several subclasses exist to do this (see the conv and pgen modules). The load() method reads the tables from a pickle file, which is much faster than the other ways offered by subclasses. The pickle file is written by calling dump() (after loading the grammar tables using a subclass). The report() method prints a readable representation of the tables to stdout, for debugging. The instance variables are as follows: symbol2number -- a dict mapping symbol names to numbers. Symbol numbers are always 256 or higher, to distinguish them from token numbers, which are between 0 and 255 (inclusive). number2symbol -- a dict mapping numbers to symbol names; these two are each other's inverse. states -- a list of DFAs, where each DFA is a list of states, each state is a list of arcs, and each arc is a (i, j) pair where i is a label and j is a state number. The DFA number is the index into this list. (This name is slightly confusing.) Final states are represented by a special arc of the form (0, j) where j is its own state number. dfas -- a dict mapping symbol numbers to (DFA, first) pairs, where DFA is an item from the states list above, and first is a set of tokens that can begin this grammar rule (represented by a dict whose values are always 1). labels -- a list of (x, y) pairs where x is either a token number or a symbol number, and y is either None or a string; the strings are keywords. The label number is the index in this list; label numbers are used to mark state transitions (arcs) in the DFAs. start -- the number of the grammar's start symbol. keywords -- a dict mapping keyword strings to arc labels. tokens -- a dict mapping token numbers to arc labels. """ def __init__(self): self.symbol2number = {} self.number2symbol = {} self.states = [] self.dfas = {} self.labels = [(0, "EMPTY")] self.keywords = {} self.tokens = {} self.symbol2label = {} self.start = 256 def dump(self, filename): """Dump the grammar tables to a pickle file. dump() recursively changes all dict to OrderedDict, so the pickled file is not exactly the same as what was passed in to dump(). load() uses the pickled file to create the tables, but only changes OrderedDict to dict at the top level; it does not recursively change OrderedDict to dict. So, the loaded tables are different from the original tables that were passed to load() in that some of the OrderedDict (from the pickled file) are not changed back to dict. For parsing, this has no effect on performance because OrderedDict uses dict's __getitem__ with nothing in between. """ with open(filename, "wb") as f: d = _make_deterministic(self.__dict__) pickle.dump(d, f, 2) def load(self, filename): """Load the grammar tables from a pickle file.""" with open(filename, "rb") as f: d = pickle.load(f) self.__dict__.update(d) def loads(self, pkl): """Load the grammar tables from a pickle bytes object.""" self.__dict__.update(pickle.loads(pkl)) def copy(self): """ Copy the grammar. """ new = self.__class__() for dict_attr in ("symbol2number", "number2symbol", "dfas", "keywords", "tokens", "symbol2label"): setattr(new, dict_attr, getattr(self, dict_attr).copy()) new.labels = self.labels[:] new.states = self.states[:] new.start = self.start return new def report(self): """Dump the grammar tables to standard output, for debugging.""" from pprint import pprint print("s2n") pprint(self.symbol2number) print("n2s") pprint(self.number2symbol) print("states") pprint(self.states) print("dfas") pprint(self.dfas) print("labels") pprint(self.labels) print("start", self.start) def _make_deterministic(top): if isinstance(top, dict): return collections.OrderedDict( sorted(((k, _make_deterministic(v)) for k, v in top.items()))) if isinstance(top, list): return [_make_deterministic(e) for e in top] if isinstance(top, tuple): return tuple(_make_deterministic(e) for e in top) return top # Map from operator to number (since tokenize doesn't do this) opmap_raw = """ ( LPAR ) RPAR [ LSQB ] RSQB : COLON , COMMA ; SEMI + PLUS - MINUS * STAR / SLASH | VBAR & AMPER < LESS > GREATER = EQUAL . DOT % PERCENT ` BACKQUOTE { LBRACE } RBRACE @ AT @= ATEQUAL == EQEQUAL != NOTEQUAL <> NOTEQUAL <= LESSEQUAL >= GREATEREQUAL ~ TILDE ^ CIRCUMFLEX << LEFTSHIFT >> RIGHTSHIFT ** DOUBLESTAR += PLUSEQUAL -= MINEQUAL *= STAREQUAL /= SLASHEQUAL %= PERCENTEQUAL &= AMPEREQUAL |= VBAREQUAL ^= CIRCUMFLEXEQUAL <<= LEFTSHIFTEQUAL >>= RIGHTSHIFTEQUAL **= DOUBLESTAREQUAL // DOUBLESLASH //= DOUBLESLASHEQUAL -> RARROW """ opmap = {} for line in opmap_raw.splitlines(): if line: op, name = line.split() opmap[op] = getattr(token, name) conv.py000064400000022652152527262410006100 0ustar00# Copyright 2004-2005 Elemental Security, Inc. All Rights Reserved. # Licensed to PSF under a Contributor Agreement. """Convert graminit.[ch] spit out by pgen to Python code. Pgen is the Python parser generator. It is useful to quickly create a parser from a grammar file in Python's grammar notation. But I don't want my parsers to be written in C (yet), so I'm translating the parsing tables to Python data structures and writing a Python parse engine. Note that the token numbers are constants determined by the standard Python tokenizer. The standard token module defines these numbers and their names (the names are not used much). The token numbers are hardcoded into the Python tokenizer and into pgen. A Python implementation of the Python tokenizer is also available, in the standard tokenize module. On the other hand, symbol numbers (representing the grammar's non-terminals) are assigned by pgen based on the actual grammar input. Note: this module is pretty much obsolete; the pgen module generates equivalent grammar tables directly from the Grammar.txt input file without having to invoke the Python pgen C program. """ # Python imports import re # Local imports from pgen2 import grammar, token class Converter(grammar.Grammar): """Grammar subclass that reads classic pgen output files. The run() method reads the tables as produced by the pgen parser generator, typically contained in two C files, graminit.h and graminit.c. The other methods are for internal use only. See the base class for more documentation. """ def run(self, graminit_h, graminit_c): """Load the grammar tables from the text files written by pgen.""" self.parse_graminit_h(graminit_h) self.parse_graminit_c(graminit_c) self.finish_off() def parse_graminit_h(self, filename): """Parse the .h file written by pgen. (Internal) This file is a sequence of #define statements defining the nonterminals of the grammar as numbers. We build two tables mapping the numbers to names and back. """ try: f = open(filename) except OSError as err: print("Can't open %s: %s" % (filename, err)) return False self.symbol2number = {} self.number2symbol = {} lineno = 0 for line in f: lineno += 1 mo = re.match(r"^#define\s+(\w+)\s+(\d+)$", line) if not mo and line.strip(): print("%s(%s): can't parse %s" % (filename, lineno, line.strip())) else: symbol, number = mo.groups() number = int(number) assert symbol not in self.symbol2number assert number not in self.number2symbol self.symbol2number[symbol] = number self.number2symbol[number] = symbol return True def parse_graminit_c(self, filename): """Parse the .c file written by pgen. (Internal) The file looks as follows. The first two lines are always this: #include "pgenheaders.h" #include "grammar.h" After that come four blocks: 1) one or more state definitions 2) a table defining dfas 3) a table defining labels 4) a struct defining the grammar A state definition has the following form: - one or more arc arrays, each of the form: static arc arcs__[] = { {, }, ... }; - followed by a state array, of the form: static state states_[] = { {, arcs__}, ... }; """ try: f = open(filename) except OSError as err: print("Can't open %s: %s" % (filename, err)) return False # The code below essentially uses f's iterator-ness! lineno = 0 # Expect the two #include lines lineno, line = lineno+1, next(f) assert line == '#include "pgenheaders.h"\n', (lineno, line) lineno, line = lineno+1, next(f) assert line == '#include "grammar.h"\n', (lineno, line) # Parse the state definitions lineno, line = lineno+1, next(f) allarcs = {} states = [] while line.startswith("static arc "): while line.startswith("static arc "): mo = re.match(r"static arc arcs_(\d+)_(\d+)\[(\d+)\] = {$", line) assert mo, (lineno, line) n, m, k = list(map(int, mo.groups())) arcs = [] for _ in range(k): lineno, line = lineno+1, next(f) mo = re.match(r"\s+{(\d+), (\d+)},$", line) assert mo, (lineno, line) i, j = list(map(int, mo.groups())) arcs.append((i, j)) lineno, line = lineno+1, next(f) assert line == "};\n", (lineno, line) allarcs[(n, m)] = arcs lineno, line = lineno+1, next(f) mo = re.match(r"static state states_(\d+)\[(\d+)\] = {$", line) assert mo, (lineno, line) s, t = list(map(int, mo.groups())) assert s == len(states), (lineno, line) state = [] for _ in range(t): lineno, line = lineno+1, next(f) mo = re.match(r"\s+{(\d+), arcs_(\d+)_(\d+)},$", line) assert mo, (lineno, line) k, n, m = list(map(int, mo.groups())) arcs = allarcs[n, m] assert k == len(arcs), (lineno, line) state.append(arcs) states.append(state) lineno, line = lineno+1, next(f) assert line == "};\n", (lineno, line) lineno, line = lineno+1, next(f) self.states = states # Parse the dfas dfas = {} mo = re.match(r"static dfa dfas\[(\d+)\] = {$", line) assert mo, (lineno, line) ndfas = int(mo.group(1)) for i in range(ndfas): lineno, line = lineno+1, next(f) mo = re.match(r'\s+{(\d+), "(\w+)", (\d+), (\d+), states_(\d+),$', line) assert mo, (lineno, line) symbol = mo.group(2) number, x, y, z = list(map(int, mo.group(1, 3, 4, 5))) assert self.symbol2number[symbol] == number, (lineno, line) assert self.number2symbol[number] == symbol, (lineno, line) assert x == 0, (lineno, line) state = states[z] assert y == len(state), (lineno, line) lineno, line = lineno+1, next(f) mo = re.match(r'\s+("(?:\\\d\d\d)*")},$', line) assert mo, (lineno, line) first = {} rawbitset = eval(mo.group(1)) for i, c in enumerate(rawbitset): byte = ord(c) for j in range(8): if byte & (1<>=?", r"<<=?", r"<>", r"!=", r"//=?", r"->", r"[+\-*/%&@|^=<>]=?", r"~") Bracket = '[][(){}]' Special = group(r'\r?\n', r'[:;.,`@]') Funny = group(Operator, Bracket, Special) PlainToken = group(Number, Funny, String, Name) Token = Ignore + PlainToken # First (or only) line of ' or " string. ContStr = group(_litprefix + r"'[^\n'\\]*(?:\\.[^\n'\\]*)*" + group("'", r'\\\r?\n'), _litprefix + r'"[^\n"\\]*(?:\\.[^\n"\\]*)*' + group('"', r'\\\r?\n')) PseudoExtras = group(r'\\\r?\n', Comment, Triple) PseudoToken = Whitespace + group(PseudoExtras, Number, Funny, ContStr, Name) tokenprog, pseudoprog, single3prog, double3prog = list(map( re.compile, (Token, PseudoToken, Single3, Double3))) endprogs = {"'": re.compile(Single), '"': re.compile(Double), "'''": single3prog, '"""': double3prog, "r'''": single3prog, 'r"""': double3prog, "u'''": single3prog, 'u"""': double3prog, "b'''": single3prog, 'b"""': double3prog, "f'''": single3prog, 'f"""': double3prog, "ur'''": single3prog, 'ur"""': double3prog, "br'''": single3prog, 'br"""': double3prog, "rb'''": single3prog, 'rb"""': double3prog, "R'''": single3prog, 'R"""': double3prog, "U'''": single3prog, 'U"""': double3prog, "B'''": single3prog, 'B"""': double3prog, "F'''": single3prog, 'F"""': double3prog, "uR'''": single3prog, 'uR"""': double3prog, "Ur'''": single3prog, 'Ur"""': double3prog, "UR'''": single3prog, 'UR"""': double3prog, "bR'''": single3prog, 'bR"""': double3prog, "Br'''": single3prog, 'Br"""': double3prog, "BR'''": single3prog, 'BR"""': double3prog, "rB'''": single3prog, 'rB"""': double3prog, "Rb'''": single3prog, 'Rb"""': double3prog, "RB'''": single3prog, 'RB"""': double3prog, 'r': None, 'R': None, 'u': None, 'U': None, 'f': None, 'F': None, 'b': None, 'B': None} triple_quoted = {} for t in ("'''", '"""', "r'''", 'r"""', "R'''", 'R"""', "u'''", 'u"""', "U'''", 'U"""', "b'''", 'b"""', "B'''", 'B"""', "f'''", 'f"""', "F'''", 'F"""', "ur'''", 'ur"""', "Ur'''", 'Ur"""', "uR'''", 'uR"""', "UR'''", 'UR"""', "br'''", 'br"""', "Br'''", 'Br"""', "bR'''", 'bR"""', "BR'''", 'BR"""', "rb'''", 'rb"""', "Rb'''", 'Rb"""', "rB'''", 'rB"""', "RB'''", 'RB"""',): triple_quoted[t] = t single_quoted = {} for t in ("'", '"', "r'", 'r"', "R'", 'R"', "u'", 'u"', "U'", 'U"', "b'", 'b"', "B'", 'B"', "f'", 'f"', "F'", 'F"', "ur'", 'ur"', "Ur'", 'Ur"', "uR'", 'uR"', "UR'", 'UR"', "br'", 'br"', "Br'", 'Br"', "bR'", 'bR"', "BR'", 'BR"', "rb'", 'rb"', "Rb'", 'Rb"', "rB'", 'rB"', "RB'", 'RB"',): single_quoted[t] = t tabsize = 8 class TokenError(Exception): pass class StopTokenizing(Exception): pass def printtoken(type, token, xxx_todo_changeme, xxx_todo_changeme1, line): # for testing (srow, scol) = xxx_todo_changeme (erow, ecol) = xxx_todo_changeme1 print("%d,%d-%d,%d:\t%s\t%s" % \ (srow, scol, erow, ecol, tok_name[type], repr(token))) def tokenize(readline, tokeneater=printtoken): """ The tokenize() function accepts two parameters: one representing the input stream, and one providing an output mechanism for tokenize(). The first parameter, readline, must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string. The second parameter, tokeneater, must also be a callable object. It is called once for each token, with five arguments, corresponding to the tuples generated by generate_tokens(). """ try: tokenize_loop(readline, tokeneater) except StopTokenizing: pass # backwards compatible interface def tokenize_loop(readline, tokeneater): for token_info in generate_tokens(readline): tokeneater(*token_info) class Untokenizer: def __init__(self): self.tokens = [] self.prev_row = 1 self.prev_col = 0 def add_whitespace(self, start): row, col = start assert row <= self.prev_row col_offset = col - self.prev_col if col_offset: self.tokens.append(" " * col_offset) def untokenize(self, iterable): for t in iterable: if len(t) == 2: self.compat(t, iterable) break tok_type, token, start, end, line = t self.add_whitespace(start) self.tokens.append(token) self.prev_row, self.prev_col = end if tok_type in (NEWLINE, NL): self.prev_row += 1 self.prev_col = 0 return "".join(self.tokens) def compat(self, token, iterable): startline = False indents = [] toks_append = self.tokens.append toknum, tokval = token if toknum in (NAME, NUMBER): tokval += ' ' if toknum in (NEWLINE, NL): startline = True for tok in iterable: toknum, tokval = tok[:2] if toknum in (NAME, NUMBER, ASYNC, AWAIT): tokval += ' ' if toknum == INDENT: indents.append(tokval) continue elif toknum == DEDENT: indents.pop() continue elif toknum in (NEWLINE, NL): startline = True elif startline and indents: toks_append(indents[-1]) startline = False toks_append(tokval) cookie_re = re.compile(r'^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)', re.ASCII) blank_re = re.compile(br'^[ \t\f]*(?:[#\r\n]|$)', re.ASCII) def _get_normal_name(orig_enc): """Imitates get_normal_name in tokenizer.c.""" # Only care about the first 12 characters. enc = orig_enc[:12].lower().replace("_", "-") if enc == "utf-8" or enc.startswith("utf-8-"): return "utf-8" if enc in ("latin-1", "iso-8859-1", "iso-latin-1") or \ enc.startswith(("latin-1-", "iso-8859-1-", "iso-latin-1-")): return "iso-8859-1" return orig_enc def detect_encoding(readline): """ The detect_encoding() function is used to detect the encoding that should be used to decode a Python source file. It requires one argument, readline, in the same way as the tokenize() generator. It will call readline a maximum of twice, and return the encoding used (as a string) and a list of any lines (left as bytes) it has read in. It detects the encoding from the presence of a utf-8 bom or an encoding cookie as specified in pep-0263. If both a bom and a cookie are present, but disagree, a SyntaxError will be raised. If the encoding cookie is an invalid charset, raise a SyntaxError. Note that if a utf-8 bom is found, 'utf-8-sig' is returned. If no encoding is specified, then the default of 'utf-8' will be returned. """ bom_found = False encoding = None default = 'utf-8' def read_or_stop(): try: return readline() except StopIteration: return bytes() def find_cookie(line): try: line_string = line.decode('ascii') except UnicodeDecodeError: return None match = cookie_re.match(line_string) if not match: return None encoding = _get_normal_name(match.group(1)) try: codec = lookup(encoding) except LookupError: # This behaviour mimics the Python interpreter raise SyntaxError("unknown encoding: " + encoding) if bom_found: if codec.name != 'utf-8': # This behaviour mimics the Python interpreter raise SyntaxError('encoding problem: utf-8') encoding += '-sig' return encoding first = read_or_stop() if first.startswith(BOM_UTF8): bom_found = True first = first[3:] default = 'utf-8-sig' if not first: return default, [] encoding = find_cookie(first) if encoding: return encoding, [first] if not blank_re.match(first): return default, [first] second = read_or_stop() if not second: return default, [first] encoding = find_cookie(second) if encoding: return encoding, [first, second] return default, [first, second] def untokenize(iterable): """Transform tokens back into Python source code. Each element returned by the iterable must be a token sequence with at least two elements, a token number and token value. If only two tokens are passed, the resulting output is poor. Round-trip invariant for full input: Untokenized source will match input source exactly Round-trip invariant for limited intput: # Output text will tokenize the back to the input t1 = [tok[:2] for tok in generate_tokens(f.readline)] newcode = untokenize(t1) readline = iter(newcode.splitlines(1)).next t2 = [tok[:2] for tokin generate_tokens(readline)] assert t1 == t2 """ ut = Untokenizer() return ut.untokenize(iterable) def generate_tokens(readline): """ The generate_tokens() generator requires one argument, readline, which must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string. Alternately, readline can be a callable function terminating with StopIteration: readline = open(myfile).next # Example of alternate readline The generator produces 5-tuples with these members: the token type; the token string; a 2-tuple (srow, scol) of ints specifying the row and column where the token begins in the source; a 2-tuple (erow, ecol) of ints specifying the row and column where the token ends in the source; and the line on which the token was found. The line passed is the logical line; continuation lines are included. """ lnum = parenlev = continued = 0 namechars, numchars = string.ascii_letters + '_', '0123456789' contstr, needcont = '', 0 contline = None indents = [0] # 'stashed' and 'async_*' are used for async/await parsing stashed = None async_def = False async_def_indent = 0 async_def_nl = False while 1: # loop over lines in stream try: line = readline() except StopIteration: line = '' lnum = lnum + 1 pos, max = 0, len(line) if contstr: # continued string if not line: raise TokenError("EOF in multi-line string", strstart) endmatch = endprog.match(line) if endmatch: pos = end = endmatch.end(0) yield (STRING, contstr + line[:end], strstart, (lnum, end), contline + line) contstr, needcont = '', 0 contline = None elif needcont and line[-2:] != '\\\n' and line[-3:] != '\\\r\n': yield (ERRORTOKEN, contstr + line, strstart, (lnum, len(line)), contline) contstr = '' contline = None continue else: contstr = contstr + line contline = contline + line continue elif parenlev == 0 and not continued: # new statement if not line: break column = 0 while pos < max: # measure leading whitespace if line[pos] == ' ': column = column + 1 elif line[pos] == '\t': column = (column//tabsize + 1)*tabsize elif line[pos] == '\f': column = 0 else: break pos = pos + 1 if pos == max: break if stashed: yield stashed stashed = None if line[pos] in '#\r\n': # skip comments or blank lines if line[pos] == '#': comment_token = line[pos:].rstrip('\r\n') nl_pos = pos + len(comment_token) yield (COMMENT, comment_token, (lnum, pos), (lnum, pos + len(comment_token)), line) yield (NL, line[nl_pos:], (lnum, nl_pos), (lnum, len(line)), line) else: yield ((NL, COMMENT)[line[pos] == '#'], line[pos:], (lnum, pos), (lnum, len(line)), line) continue if column > indents[-1]: # count indents or dedents indents.append(column) yield (INDENT, line[:pos], (lnum, 0), (lnum, pos), line) while column < indents[-1]: if column not in indents: raise IndentationError( "unindent does not match any outer indentation level", ("", lnum, pos, line)) indents = indents[:-1] if async_def and async_def_indent >= indents[-1]: async_def = False async_def_nl = False async_def_indent = 0 yield (DEDENT, '', (lnum, pos), (lnum, pos), line) if async_def and async_def_nl and async_def_indent >= indents[-1]: async_def = False async_def_nl = False async_def_indent = 0 else: # continued statement if not line: raise TokenError("EOF in multi-line statement", (lnum, 0)) continued = 0 while pos < max: pseudomatch = pseudoprog.match(line, pos) if pseudomatch: # scan for tokens start, end = pseudomatch.span(1) spos, epos, pos = (lnum, start), (lnum, end), end token, initial = line[start:end], line[start] if initial in numchars or \ (initial == '.' and token != '.'): # ordinary number yield (NUMBER, token, spos, epos, line) elif initial in '\r\n': newline = NEWLINE if parenlev > 0: newline = NL elif async_def: async_def_nl = True if stashed: yield stashed stashed = None yield (newline, token, spos, epos, line) elif initial == '#': assert not token.endswith("\n") if stashed: yield stashed stashed = None yield (COMMENT, token, spos, epos, line) elif token in triple_quoted: endprog = endprogs[token] endmatch = endprog.match(line, pos) if endmatch: # all on one line pos = endmatch.end(0) token = line[start:pos] if stashed: yield stashed stashed = None yield (STRING, token, spos, (lnum, pos), line) else: strstart = (lnum, start) # multiple lines contstr = line[start:] contline = line break elif initial in single_quoted or \ token[:2] in single_quoted or \ token[:3] in single_quoted: if token[-1] == '\n': # continued string strstart = (lnum, start) endprog = (endprogs[initial] or endprogs[token[1]] or endprogs[token[2]]) contstr, needcont = line[start:], 1 contline = line break else: # ordinary string if stashed: yield stashed stashed = None yield (STRING, token, spos, epos, line) elif initial in namechars: # ordinary name if token in ('async', 'await'): if async_def: yield (ASYNC if token == 'async' else AWAIT, token, spos, epos, line) continue tok = (NAME, token, spos, epos, line) if token == 'async' and not stashed: stashed = tok continue if token == 'def': if (stashed and stashed[0] == NAME and stashed[1] == 'async'): async_def = True async_def_indent = indents[-1] yield (ASYNC, stashed[1], stashed[2], stashed[3], stashed[4]) stashed = None if stashed: yield stashed stashed = None yield tok elif initial == '\\': # continued stmt # This yield is new; needed for better idempotency: if stashed: yield stashed stashed = None yield (NL, token, spos, (lnum, pos), line) continued = 1 else: if initial in '([{': parenlev = parenlev + 1 elif initial in ')]}': parenlev = parenlev - 1 if stashed: yield stashed stashed = None yield (OP, token, spos, epos, line) else: yield (ERRORTOKEN, line[pos], (lnum, pos), (lnum, pos+1), line) pos = pos + 1 if stashed: yield stashed stashed = None for indent in indents[1:]: # pop remaining indent levels yield (DEDENT, '', (lnum, 0), (lnum, 0), '') yield (ENDMARKER, '', (lnum, 0), (lnum, 0), '') if __name__ == '__main__': # testing import sys if len(sys.argv) > 1: tokenize(open(sys.argv[1]).readline) else: tokenize(sys.stdin.readline) pgen.py000064400000032764152527262410006071 0ustar00# Copyright 2004-2005 Elemental Security, Inc. All Rights Reserved. # Licensed to PSF under a Contributor Agreement. # Pgen imports from . import grammar, token, tokenize class PgenGrammar(grammar.Grammar): pass class ParserGenerator(object): def __init__(self, filename, stream=None): close_stream = None if stream is None: stream = open(filename) close_stream = stream.close self.filename = filename self.stream = stream self.generator = tokenize.generate_tokens(stream.readline) self.gettoken() # Initialize lookahead self.dfas, self.startsymbol = self.parse() if close_stream is not None: close_stream() self.first = {} # map from symbol name to set of tokens self.addfirstsets() def make_grammar(self): c = PgenGrammar() names = list(self.dfas.keys()) names.sort() names.remove(self.startsymbol) names.insert(0, self.startsymbol) for name in names: i = 256 + len(c.symbol2number) c.symbol2number[name] = i c.number2symbol[i] = name for name in names: dfa = self.dfas[name] states = [] for state in dfa: arcs = [] for label, next in sorted(state.arcs.items()): arcs.append((self.make_label(c, label), dfa.index(next))) if state.isfinal: arcs.append((0, dfa.index(state))) states.append(arcs) c.states.append(states) c.dfas[c.symbol2number[name]] = (states, self.make_first(c, name)) c.start = c.symbol2number[self.startsymbol] return c def make_first(self, c, name): rawfirst = self.first[name] first = {} for label in sorted(rawfirst): ilabel = self.make_label(c, label) ##assert ilabel not in first # XXX failed on <> ... != first[ilabel] = 1 return first def make_label(self, c, label): # XXX Maybe this should be a method on a subclass of converter? ilabel = len(c.labels) if label[0].isalpha(): # Either a symbol name or a named token if label in c.symbol2number: # A symbol name (a non-terminal) if label in c.symbol2label: return c.symbol2label[label] else: c.labels.append((c.symbol2number[label], None)) c.symbol2label[label] = ilabel return ilabel else: # A named token (NAME, NUMBER, STRING) itoken = getattr(token, label, None) assert isinstance(itoken, int), label assert itoken in token.tok_name, label if itoken in c.tokens: return c.tokens[itoken] else: c.labels.append((itoken, None)) c.tokens[itoken] = ilabel return ilabel else: # Either a keyword or an operator assert label[0] in ('"', "'"), label value = eval(label) if value[0].isalpha(): # A keyword if value in c.keywords: return c.keywords[value] else: c.labels.append((token.NAME, value)) c.keywords[value] = ilabel return ilabel else: # An operator (any non-numeric token) itoken = grammar.opmap[value] # Fails if unknown token if itoken in c.tokens: return c.tokens[itoken] else: c.labels.append((itoken, None)) c.tokens[itoken] = ilabel return ilabel def addfirstsets(self): names = list(self.dfas.keys()) names.sort() for name in names: if name not in self.first: self.calcfirst(name) #print name, self.first[name].keys() def calcfirst(self, name): dfa = self.dfas[name] self.first[name] = None # dummy to detect left recursion state = dfa[0] totalset = {} overlapcheck = {} for label, next in state.arcs.items(): if label in self.dfas: if label in self.first: fset = self.first[label] if fset is None: raise ValueError("recursion for rule %r" % name) else: self.calcfirst(label) fset = self.first[label] totalset.update(fset) overlapcheck[label] = fset else: totalset[label] = 1 overlapcheck[label] = {label: 1} inverse = {} for label, itsfirst in overlapcheck.items(): for symbol in itsfirst: if symbol in inverse: raise ValueError("rule %s is ambiguous; %s is in the" " first sets of %s as well as %s" % (name, symbol, label, inverse[symbol])) inverse[symbol] = label self.first[name] = totalset def parse(self): dfas = {} startsymbol = None # MSTART: (NEWLINE | RULE)* ENDMARKER while self.type != token.ENDMARKER: while self.type == token.NEWLINE: self.gettoken() # RULE: NAME ':' RHS NEWLINE name = self.expect(token.NAME) self.expect(token.OP, ":") a, z = self.parse_rhs() self.expect(token.NEWLINE) #self.dump_nfa(name, a, z) dfa = self.make_dfa(a, z) #self.dump_dfa(name, dfa) oldlen = len(dfa) self.simplify_dfa(dfa) newlen = len(dfa) dfas[name] = dfa #print name, oldlen, newlen if startsymbol is None: startsymbol = name return dfas, startsymbol def make_dfa(self, start, finish): # To turn an NFA into a DFA, we define the states of the DFA # to correspond to *sets* of states of the NFA. Then do some # state reduction. Let's represent sets as dicts with 1 for # values. assert isinstance(start, NFAState) assert isinstance(finish, NFAState) def closure(state): base = {} addclosure(state, base) return base def addclosure(state, base): assert isinstance(state, NFAState) if state in base: return base[state] = 1 for label, next in state.arcs: if label is None: addclosure(next, base) states = [DFAState(closure(start), finish)] for state in states: # NB states grows while we're iterating arcs = {} for nfastate in state.nfaset: for label, next in nfastate.arcs: if label is not None: addclosure(next, arcs.setdefault(label, {})) for label, nfaset in sorted(arcs.items()): for st in states: if st.nfaset == nfaset: break else: st = DFAState(nfaset, finish) states.append(st) state.addarc(st, label) return states # List of DFAState instances; first one is start def dump_nfa(self, name, start, finish): print("Dump of NFA for", name) todo = [start] for i, state in enumerate(todo): print(" State", i, state is finish and "(final)" or "") for label, next in state.arcs: if next in todo: j = todo.index(next) else: j = len(todo) todo.append(next) if label is None: print(" -> %d" % j) else: print(" %s -> %d" % (label, j)) def dump_dfa(self, name, dfa): print("Dump of DFA for", name) for i, state in enumerate(dfa): print(" State", i, state.isfinal and "(final)" or "") for label, next in sorted(state.arcs.items()): print(" %s -> %d" % (label, dfa.index(next))) def simplify_dfa(self, dfa): # This is not theoretically optimal, but works well enough. # Algorithm: repeatedly look for two states that have the same # set of arcs (same labels pointing to the same nodes) and # unify them, until things stop changing. # dfa is a list of DFAState instances changes = True while changes: changes = False for i, state_i in enumerate(dfa): for j in range(i+1, len(dfa)): state_j = dfa[j] if state_i == state_j: #print " unify", i, j del dfa[j] for state in dfa: state.unifystate(state_j, state_i) changes = True break def parse_rhs(self): # RHS: ALT ('|' ALT)* a, z = self.parse_alt() if self.value != "|": return a, z else: aa = NFAState() zz = NFAState() aa.addarc(a) z.addarc(zz) while self.value == "|": self.gettoken() a, z = self.parse_alt() aa.addarc(a) z.addarc(zz) return aa, zz def parse_alt(self): # ALT: ITEM+ a, b = self.parse_item() while (self.value in ("(", "[") or self.type in (token.NAME, token.STRING)): c, d = self.parse_item() b.addarc(c) b = d return a, b def parse_item(self): # ITEM: '[' RHS ']' | ATOM ['+' | '*'] if self.value == "[": self.gettoken() a, z = self.parse_rhs() self.expect(token.OP, "]") a.addarc(z) return a, z else: a, z = self.parse_atom() value = self.value if value not in ("+", "*"): return a, z self.gettoken() z.addarc(a) if value == "+": return a, z else: return a, a def parse_atom(self): # ATOM: '(' RHS ')' | NAME | STRING if self.value == "(": self.gettoken() a, z = self.parse_rhs() self.expect(token.OP, ")") return a, z elif self.type in (token.NAME, token.STRING): a = NFAState() z = NFAState() a.addarc(z, self.value) self.gettoken() return a, z else: self.raise_error("expected (...) or NAME or STRING, got %s/%s", self.type, self.value) def expect(self, type, value=None): if self.type != type or (value is not None and self.value != value): self.raise_error("expected %s/%s, got %s/%s", type, value, self.type, self.value) value = self.value self.gettoken() return value def gettoken(self): tup = next(self.generator) while tup[0] in (tokenize.COMMENT, tokenize.NL): tup = next(self.generator) self.type, self.value, self.begin, self.end, self.line = tup #print token.tok_name[self.type], repr(self.value) def raise_error(self, msg, *args): if args: try: msg = msg % args except: msg = " ".join([msg] + list(map(str, args))) raise SyntaxError(msg, (self.filename, self.end[0], self.end[1], self.line)) class NFAState(object): def __init__(self): self.arcs = [] # list of (label, NFAState) pairs def addarc(self, next, label=None): assert label is None or isinstance(label, str) assert isinstance(next, NFAState) self.arcs.append((label, next)) class DFAState(object): def __init__(self, nfaset, final): assert isinstance(nfaset, dict) assert isinstance(next(iter(nfaset)), NFAState) assert isinstance(final, NFAState) self.nfaset = nfaset self.isfinal = final in nfaset self.arcs = {} # map from label to DFAState def addarc(self, next, label): assert isinstance(label, str) assert label not in self.arcs assert isinstance(next, DFAState) self.arcs[label] = next def unifystate(self, old, new): for label, next in self.arcs.items(): if next is old: self.arcs[label] = new def __eq__(self, other): # Equality test -- ignore the nfaset instance variable assert isinstance(other, DFAState) if self.isfinal != other.isfinal: return False # Can't just return self.arcs == other.arcs, because that # would invoke this method recursively, with cycles... if len(self.arcs) != len(other.arcs): return False for label, next in self.arcs.items(): if next is not other.arcs.get(label): return False return True __hash__ = None # For Py3 compatibility. def generate_grammar(filename="Grammar.txt"): p = ParserGenerator(filename) return p.make_grammar() __init__.py000064400000000217152527262410006663 0ustar00# Copyright 2004-2005 Elemental Security, Inc. All Rights Reserved. # Licensed to PSF under a Contributor Agreement. """The pgen2 package.""" driver.py000064400000013614152527262410006424 0ustar00# Copyright 2004-2005 Elemental Security, Inc. All Rights Reserved. # Licensed to PSF under a Contributor Agreement. # Modifications: # Copyright 2006 Google, Inc. All Rights Reserved. # Licensed to PSF under a Contributor Agreement. """Parser driver. This provides a high-level interface to parse a file into a syntax tree. """ __author__ = "Guido van Rossum " __all__ = ["Driver", "load_grammar"] # Python imports import codecs import io import os import logging import pkgutil import sys # Pgen imports from . import grammar, parse, token, tokenize, pgen class Driver(object): def __init__(self, grammar, convert=None, logger=None): self.grammar = grammar if logger is None: logger = logging.getLogger() self.logger = logger self.convert = convert def parse_tokens(self, tokens, debug=False): """Parse a series of tokens and return the syntax tree.""" # XXX Move the prefix computation into a wrapper around tokenize. p = parse.Parser(self.grammar, self.convert) p.setup() lineno = 1 column = 0 type = value = start = end = line_text = None prefix = "" for quintuple in tokens: type, value, start, end, line_text = quintuple if start != (lineno, column): assert (lineno, column) <= start, ((lineno, column), start) s_lineno, s_column = start if lineno < s_lineno: prefix += "\n" * (s_lineno - lineno) lineno = s_lineno column = 0 if column < s_column: prefix += line_text[column:s_column] column = s_column if type in (tokenize.COMMENT, tokenize.NL): prefix += value lineno, column = end if value.endswith("\n"): lineno += 1 column = 0 continue if type == token.OP: type = grammar.opmap[value] if debug: self.logger.debug("%s %r (prefix=%r)", token.tok_name[type], value, prefix) if p.addtoken(type, value, (prefix, start)): if debug: self.logger.debug("Stop.") break prefix = "" lineno, column = end if value.endswith("\n"): lineno += 1 column = 0 else: # We never broke out -- EOF is too soon (how can this happen???) raise parse.ParseError("incomplete input", type, value, (prefix, start)) return p.rootnode def parse_stream_raw(self, stream, debug=False): """Parse a stream and return the syntax tree.""" tokens = tokenize.generate_tokens(stream.readline) return self.parse_tokens(tokens, debug) def parse_stream(self, stream, debug=False): """Parse a stream and return the syntax tree.""" return self.parse_stream_raw(stream, debug) def parse_file(self, filename, encoding=None, debug=False): """Parse a file and return the syntax tree.""" stream = codecs.open(filename, "r", encoding) try: return self.parse_stream(stream, debug) finally: stream.close() def parse_string(self, text, debug=False): """Parse a string and return the syntax tree.""" tokens = tokenize.generate_tokens(io.StringIO(text).readline) return self.parse_tokens(tokens, debug) def _generate_pickle_name(gt): head, tail = os.path.splitext(gt) if tail == ".txt": tail = "" return head + tail + ".".join(map(str, sys.version_info)) + ".pickle" def load_grammar(gt="Grammar.txt", gp=None, save=True, force=False, logger=None): """Load the grammar (maybe from a pickle).""" if logger is None: logger = logging.getLogger() gp = _generate_pickle_name(gt) if gp is None else gp if force or not _newer(gp, gt): logger.info("Generating grammar tables from %s", gt) g = pgen.generate_grammar(gt) if save: logger.info("Writing grammar tables to %s", gp) try: g.dump(gp) except OSError as e: logger.info("Writing failed: %s", e) else: g = grammar.Grammar() g.load(gp) return g def _newer(a, b): """Inquire whether file a was written since file b.""" if not os.path.exists(a): return False if not os.path.exists(b): return True return os.path.getmtime(a) >= os.path.getmtime(b) def load_packaged_grammar(package, grammar_source): """Normally, loads a pickled grammar by doing pkgutil.get_data(package, pickled_grammar) where *pickled_grammar* is computed from *grammar_source* by adding the Python version and using a ``.pickle`` extension. However, if *grammar_source* is an extant file, load_grammar(grammar_source) is called instead. This facilitates using a packaged grammar file when needed but preserves load_grammar's automatic regeneration behavior when possible. """ if os.path.isfile(grammar_source): return load_grammar(grammar_source) pickled_name = _generate_pickle_name(os.path.basename(grammar_source)) data = pkgutil.get_data(package, pickled_name) g = grammar.Grammar() g.loads(data) return g def main(*args): """Main program, when run as a script: produce grammar pickle files. Calls load_grammar for each argument, a path to a grammar text file. """ if not args: args = sys.argv[1:] logging.basicConfig(level=logging.INFO, stream=sys.stdout, format='%(message)s') for gt in args: load_grammar(gt, save=True, force=True) return True if __name__ == "__main__": sys.exit(int(not main())) token.py000075500000002416152527262410006252 0ustar00#! /usr/libexec/platform-python """Token constants (from "token.h").""" # Taken from Python (r53757) and modified to include some tokens # originally monkeypatched in by pgen2.tokenize #--start constants-- ENDMARKER = 0 NAME = 1 NUMBER = 2 STRING = 3 NEWLINE = 4 INDENT = 5 DEDENT = 6 LPAR = 7 RPAR = 8 LSQB = 9 RSQB = 10 COLON = 11 COMMA = 12 SEMI = 13 PLUS = 14 MINUS = 15 STAR = 16 SLASH = 17 VBAR = 18 AMPER = 19 LESS = 20 GREATER = 21 EQUAL = 22 DOT = 23 PERCENT = 24 BACKQUOTE = 25 LBRACE = 26 RBRACE = 27 EQEQUAL = 28 NOTEQUAL = 29 LESSEQUAL = 30 GREATEREQUAL = 31 TILDE = 32 CIRCUMFLEX = 33 LEFTSHIFT = 34 RIGHTSHIFT = 35 DOUBLESTAR = 36 PLUSEQUAL = 37 MINEQUAL = 38 STAREQUAL = 39 SLASHEQUAL = 40 PERCENTEQUAL = 41 AMPEREQUAL = 42 VBAREQUAL = 43 CIRCUMFLEXEQUAL = 44 LEFTSHIFTEQUAL = 45 RIGHTSHIFTEQUAL = 46 DOUBLESTAREQUAL = 47 DOUBLESLASH = 48 DOUBLESLASHEQUAL = 49 AT = 50 ATEQUAL = 51 OP = 52 COMMENT = 53 NL = 54 RARROW = 55 AWAIT = 56 ASYNC = 57 ERRORTOKEN = 58 N_TOKENS = 59 NT_OFFSET = 256 #--end constants-- tok_name = {} for _name, _value in list(globals().items()): if type(_value) is type(0): tok_name[_value] = _name def ISTERMINAL(x): return x < NT_OFFSET def ISNONTERMINAL(x): return x >= NT_OFFSET def ISEOF(x): return x == ENDMARKER __pycache__/literals.cpython-36.opt-1.pyc000064400000002502152531664540014172 0ustar003 \O @sPdZddlZddddddd d d d d ZddZddZddZedkrLedS)ztdD]2}t|}t|}t|}||kr t||||q WdS)N)rangerreprr-print)r$cr+er%r%r&test2s r4__main__)__doc__r)rr'r-r4__name__r%r%r%r&s   __pycache__/parse.cpython-36.pyc000064400000014271152531664540012534 0ustar003 \u@s4dZddlmZGdddeZGdddeZdS)zParser engine for the grammar tables generated by pgen. The grammar table must be loaded first. See Parser/parser.c in the Python distribution for additional info on how this parsing engine works. )tokenc@seZdZdZddZdS) ParseErrorz(Exception to signal the parser is stuck.cCs4tj|d||||f||_||_||_||_dS)Nz!%s: type=%r, value=%r, context=%r) Exception__init__msgtypevaluecontext)selfrrrr r +/usr/lib64/python3.6/lib2to3/pgen2/parse.pyrs zParseError.__init__N)__name__ __module__ __qualname____doc__rr r r r rsrc@sLeZdZdZdddZdddZddZd d Zd d Zd dZ ddZ dS)Parsera5Parser engine. The proper usage sequence is: p = Parser(grammar, [converter]) # create instance p.setup([start]) # prepare for parsing : if p.addtoken(...): # parse a token; may raise ParseError break root = p.rootnode # root of abstract syntax tree A Parser instance may be reused by calling setup() repeatedly. A Parser instance contains state pertaining to the current token sequence, and should not be used concurrently by different threads to parse separate token sequences. See driver.py for how to get input tokens by tokenizing a file or string. Parsing is complete when addtoken() returns True; the root of the abstract syntax tree can then be retrieved from the rootnode instance variable. When a syntax error occurs, addtoken() raises the ParseError exception. There is no error recovery; the parser cannot be used after a syntax error was reported (but it can be reinitialized by calling setup()). NcCs||_|pdd|_dS)aConstructor. The grammar argument is a grammar.Grammar instance; see the grammar module for more information. The parser is not ready yet for parsing; you must call the setup() method to get it started. The optional convert argument is a function mapping concrete syntax tree nodes to abstract syntax tree nodes. If not given, no conversion is done and the syntax tree produced is the concrete syntax tree. If given, it must be a function of two arguments, the first being the grammar (a grammar.Grammar instance), and the second being the concrete syntax tree node to be converted. The syntax tree is converted from the bottom up. A concrete syntax tree node is a (type, value, context, nodes) tuple, where type is the node type (a token or symbol number), value is None for symbols and a string for tokens, context is None or an opaque value used for error reporting (typically a (lineno, offset) pair), and nodes is a list of children for symbols, and None for tokens. An abstract syntax tree node may be anything; this is entirely up to the converter function. cSs|S)Nr )grammarnoder r r Wsz!Parser.__init__..N)rconvert)r rrr r r r9szParser.__init__cCsH|dkr|jj}|ddgf}|jj|d|f}|g|_d|_t|_dS)aPrepare for parsing. This *must* be called before starting to parse. The optional argument is an alternative start symbol; it defaults to the grammar's start symbol. You can use a Parser instance to parse any number of programs; each time you call setup() the parser is reset to an initial state determined by the (implicit or explicit) start symbol. N)rstartdfasstackrootnodeset used_names)r rnewnodeZ stackentryr r r setupYs  z Parser.setupcCsF|j|||}x0|jd \}}}|\}} ||} x| D]\} } |jj| \} }|| kr| dksft|j||| || }x@||d|fgkr|j|jsdS|jd \}}}|\}} q|WdS| dkr:|jj| }|\}}||kr:|j| |jj| | |Pq:Wd|f| kr0|j|js>t d|||qt d|||qWdS) z s  __pycache__/grammar.cpython-36.opt-1.pyc000064400000015621152531664540014007 0ustar003 \@sxdZddlZddlZddlmZmZGdddeZddZd Z iZ x.e j D]"Z e rNe j \ZZeeee e<qNWdS) aThis module defines the data structures used to represent a grammar. These are a bit arcane because they are derived from the data structures used by Python's 'pgen' parser generator. There's also a table here mapping operators to their names in the token module; the Python tokenize module reports all operators as the fallback token code OP, but the parser needs the actual token code. N)tokentokenizec@s@eZdZdZddZddZddZdd Zd d Zd d Z dS)Grammara Pgen parsing tables conversion class. Once initialized, this class supplies the grammar tables for the parsing engine implemented by parse.py. The parsing engine accesses the instance variables directly. The class here does not provide initialization of the tables; several subclasses exist to do this (see the conv and pgen modules). The load() method reads the tables from a pickle file, which is much faster than the other ways offered by subclasses. The pickle file is written by calling dump() (after loading the grammar tables using a subclass). The report() method prints a readable representation of the tables to stdout, for debugging. The instance variables are as follows: symbol2number -- a dict mapping symbol names to numbers. Symbol numbers are always 256 or higher, to distinguish them from token numbers, which are between 0 and 255 (inclusive). number2symbol -- a dict mapping numbers to symbol names; these two are each other's inverse. states -- a list of DFAs, where each DFA is a list of states, each state is a list of arcs, and each arc is a (i, j) pair where i is a label and j is a state number. The DFA number is the index into this list. (This name is slightly confusing.) Final states are represented by a special arc of the form (0, j) where j is its own state number. dfas -- a dict mapping symbol numbers to (DFA, first) pairs, where DFA is an item from the states list above, and first is a set of tokens that can begin this grammar rule (represented by a dict whose values are always 1). labels -- a list of (x, y) pairs where x is either a token number or a symbol number, and y is either None or a string; the strings are keywords. The label number is the index in this list; label numbers are used to mark state transitions (arcs) in the DFAs. start -- the number of the grammar's start symbol. keywords -- a dict mapping keyword strings to arc labels. tokens -- a dict mapping token numbers to arc labels. cCs<i|_i|_g|_i|_dg|_i|_i|_i|_d|_dS)NrEMPTY)rr) symbol2number number2symbolstatesdfaslabelskeywordstokens symbol2labelstart)selfr-/usr/lib64/python3.6/lib2to3/pgen2/grammar.py__init__MszGrammar.__init__c Cs2t|d}t|j}tj||dWdQRXdS)aDump the grammar tables to a pickle file. dump() recursively changes all dict to OrderedDict, so the pickled file is not exactly the same as what was passed in to dump(). load() uses the pickled file to create the tables, but only changes OrderedDict to dict at the top level; it does not recursively change OrderedDict to dict. So, the loaded tables are different from the original tables that were passed to load() in that some of the OrderedDict (from the pickled file) are not changed back to dict. For parsing, this has no effect on performance because OrderedDict uses dict's __getitem__ with nothing in between. wbN)open_make_deterministic__dict__pickledump)rfilenamefdrrrrXs  z Grammar.dumpc Cs0t|d}tj|}WdQRX|jj|dS)z+Load the grammar tables from a pickle file.rbN)rrloadrupdate)rrrrrrrr is z Grammar.loadcCs|jjtj|dS)z3Load the grammar tables from a pickle bytes object.N)rr!rloads)rZpklrrrr"osz Grammar.loadscCsX|j}x"dD]}t||t||jqW|jdd|_|jdd|_|j|_|S) z# Copy the grammar. rr r r rrN)rr r r rr) __class__setattrgetattrcopyr r r)rnewZ dict_attrrrrr&ssz Grammar.copycCsvddlm}td||jtd||jtd||jtd||jtd||jtd|jd S) z:Dump the grammar tables to standard output, for debugging.r)pprintZs2nZn2sr r r rN)r(printrr r r r r)rr(rrrreports      zGrammar.reportN) __name__ __module__ __qualname____doc__rrr r"r&r*rrrrrs4  rcCs^t|tr&tjtdd|jDSt|tr>dd|DSt|trZtdd|DS|S)Ncss|]\}}|t|fVqdS)N)r).0kvrrr sz&_make_deterministic..cSsg|] }t|qSr)r)r/errr sz'_make_deterministic..css|]}t|VqdS)N)r)r/r3rrrr2s) isinstancedict collections OrderedDictsorteditemslisttuple)toprrrrs   ra ( LPAR ) RPAR [ LSQB ] RSQB : COLON , COMMA ; SEMI + PLUS - MINUS * STAR / SLASH | VBAR & AMPER < LESS > GREATER = EQUAL . DOT % PERCENT ` BACKQUOTE { LBRACE } RBRACE @ AT @= ATEQUAL == EQEQUAL != NOTEQUAL <> NOTEQUAL <= LESSEQUAL >= GREATEREQUAL ~ TILDE ^ CIRCUMFLEX << LEFTSHIFT >> RIGHTSHIFT ** DOUBLESTAR += PLUSEQUAL -= MINEQUAL *= STAREQUAL /= SLASHEQUAL %= PERCENTEQUAL &= AMPEREQUAL |= VBAREQUAL ^= CIRCUMFLEXEQUAL <<= LEFTSHIFTEQUAL >>= RIGHTSHIFTEQUAL **= DOUBLESTAREQUAL // DOUBLESLASH //= DOUBLESLASHEQUAL -> RARROW )r.r7rrrobjectrrZ opmap_rawZopmap splitlineslinesplitopnamer%rrrr sy= __pycache__/driver.cpython-36.opt-1.pyc000064400000011730152531664540013651 0ustar003 \@sdZdZddgZddlZddlZddlZddlZddlZddlZddl m Z m Z m Z m Z mZGdddeZd d ZdddZddZddZddZedkrejee dS)zZParser driver. This provides a high-level interface to parse a file into a syntax tree. z#Guido van Rossum Driver load_grammarN)grammarparsetokentokenizepgenc@sHeZdZdddZdddZdddZdd d Zdd d Zdd dZdS)rNcCs&||_|dkrtj}||_||_dS)N)rlogging getLoggerloggerconvert)selfrr r r,/usr/lib64/python3.6/lib2to3/pgen2/driver.py__init__ s zDriver.__init__FcCsvtj|j|j}|jd}d}d}}}} } d} x4|D]} | \}}}} } |||fkr|\} }|| kr| d| |7} | }d}||kr| | ||7} |}|tjtjfkr| |7} | \}}|jdr@|d7}d}q@|t j krtj |}|r|j j dt j||| |j||| |fr6|r4|j j dPd} | \}}|jdr@|d7}d}q@Wtjd||| |f|jS) z4Parse a series of tokens and return the syntax tree.rrN z%s %r (prefix=%r)zStop.zincomplete input)rZParserrr ZsetuprCOMMENTNLendswithrOPZopmapr debugtok_nameZaddtokenZ ParseErrorZrootnode)rtokensrplinenocolumntypevaluestartendZ line_textprefixZ quintupleZs_linenoZs_columnrrr parse_tokens'sR      zDriver.parse_tokenscCstj|j}|j||S)z*Parse a stream and return the syntax tree.)rgenerate_tokensreadliner#)rstreamrrrrrparse_stream_rawWs zDriver.parse_stream_rawcCs |j||S)z*Parse a stream and return the syntax tree.)r')rr&rrrr parse_stream\szDriver.parse_streamc Cs*tj|d|}z |j||S|jXdS)z(Parse a file and return the syntax tree.rN)codecsopenr(close)rfilenameencodingrr&rrr parse_file`s zDriver.parse_filecCstjtj|j}|j||S)z*Parse a string and return the syntax tree.)rr$ioStringIOr%r#)rtextrrrrr parse_stringhszDriver.parse_string)NN)F)F)F)NF)F) __name__ __module__ __qualname__rr#r'r(r/r3rrrrrs   0   cCs:tjj|\}}|dkrd}||djtttjdS)Nz.txtr.z.pickle)ospathsplitextjoinmapstrsys version_info)gtheadtailrrr_generate_pickle_namensrC Grammar.txtTFcCs|dkrtj}|dkr t|n|}|s4t|| r|jd|tj|}|r|jd|y|j|Wqtk r}z|jd|WYdd}~XqXnt j }|j ||S)z'Load the grammar (maybe from a pickle).Nz!Generating grammar tables from %szWriting grammar tables to %szWriting failed: %s) r r rC_newerinfor Zgenerate_grammardumpOSErrorrGrammarload)r@Zgpsaveforcer gerrrrus     cCs8tjj|sdStjj|s dStjj|tjj|kS)z0Inquire whether file a was written since file b.FT)r8r9existsgetmtime)abrrrrEs   rEcCsFtjj|rt|Sttjj|}tj||}tj }|j ||S)aNormally, loads a pickled grammar by doing pkgutil.get_data(package, pickled_grammar) where *pickled_grammar* is computed from *grammar_source* by adding the Python version and using a ``.pickle`` extension. However, if *grammar_source* is an extant file, load_grammar(grammar_source) is called instead. This facilitates using a packaged grammar file when needed but preserves load_grammar's automatic regeneration behavior when possible. ) r8r9isfilerrCbasenamepkgutilget_datarrIloads)packageZgrammar_sourceZ pickled_namedatarMrrrload_packaged_grammars   rZcGsF|stjdd}tjtjtjddx|D]}t|dddq,WdS)zMain program, when run as a script: produce grammar pickle files. Calls load_grammar for each argument, a path to a grammar text file. rNz %(message)s)levelr&formatT)rKrL)r>argvr Z basicConfigINFOstdoutr)argsr@rrrmains  ra__main__)rDNTFN)__doc__ __author____all__r*r0r8r rUr>rrrrrr objectrrCrrErZrar4exitintrrrr s$P   __pycache__/__init__.cpython-36.opt-1.pyc000064400000000237152531664540014115 0ustar003 \@sdZdS)zThe pgen2 package.N)__doc__rr./usr/lib64/python3.6/lib2to3/pgen2/__init__.pys__pycache__/token.cpython-36.opt-1.pyc000064400000003474152531664540013504 0ustar003 \@sPdZdZdZdZdZdZdZdZdZd Z d Z d Z d Z d Z dZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZd Z d!Z!d"Z"d#Z#d$Z$d%Z%d&Z&d'Z'd(Z(d)Z)d*Z*d+Z+d,Z,d-Z-d.Z.d/Z/d0Z0d1Z1d2Z2d3Z3d4Z4d5Z5d6Z6d7Z7d8Z8d9Z9d:Z:d;Z;dx6e?e@jAD]$\ZBZCeDeCeDdkr eBe>eC<q Wd>d?ZEd@dAZFdBdCZGdDS)Ez!Token constants (from "token.h").  !"#$%&'()*+,-./0123456789:;cCs|tkS)N) NT_OFFSET)xr@+/usr/lib64/python3.6/lib2to3/pgen2/token.py ISTERMINALNsrBcCs|tkS)N)r>)r?r@r@rA ISNONTERMINALQsrCcCs|tkS)N) ENDMARKER)r?r@r@rAISEOFTsrEN)H__doc__rDNAMENUMBERSTRINGNEWLINEINDENTDEDENTLPARRPARLSQBRSQBCOLONCOMMASEMIPLUSMINUSSTARSLASHVBARAMPERLESSGREATEREQUALDOTPERCENTZ BACKQUOTELBRACERBRACEEQEQUALNOTEQUAL LESSEQUAL GREATEREQUALTILDE CIRCUMFLEX LEFTSHIFT RIGHTSHIFT DOUBLESTAR PLUSEQUALMINEQUAL STAREQUAL SLASHEQUAL PERCENTEQUAL AMPEREQUAL VBAREQUALCIRCUMFLEXEQUALLEFTSHIFTEQUALRIGHTSHIFTEQUALDOUBLESTAREQUAL DOUBLESLASHDOUBLESLASHEQUALATATEQUALOPCOMMENTNLRARROWAWAITASYNC ERRORTOKENN_TOKENSr>tok_namelistglobalsitems_nameZ_valuetyperBrCrEr@r@r@rAs__pycache__/pgen.cpython-36.opt-1.pyc000064400000022165152531664540013313 0ustar003 \5@sdddlmZmZmZGdddejZGdddeZGdddeZGdd d eZ dd d Z d S))grammartokentokenizec@s eZdZdS) PgenGrammarN)__name__ __module__ __qualname__r r */usr/lib64/python3.6/lib2to3/pgen2/pgen.pyrsrc@seZdZd&ddZddZddZdd Zd d Zd d ZddZ ddZ ddZ ddZ ddZ ddZddZddZddZd'd d!Zd"d#Zd$d%ZdS)(ParserGeneratorNcCsld}|dkrt|}|j}||_||_tj|j|_|j|j \|_ |_ |dk rZ|i|_ |j dS)N)openclosefilenamestreamrgenerate_tokensreadline generatorgettokenparsedfas startsymbolfirst addfirstsets)selfrrZ close_streamr r r __init__ szParserGenerator.__init__c Cs*t}t|jj}|j|j|j|jd|jx.|D]&}dt|j }||j |<||j |<qtt|d}||jkrz|j|S|jj|df||j|<|Snt |}|djr||j kr|j |S|jjtj |f||j |<|Sn>t j |}||jkr|j|S|jj|df||j|<|SdS)Nr)r"Zlabelsisalphar#Z symbol2labelr'getattrrtokensevalkeywordsNAMErZopmap)rr.r4r7Zitokenvaluer r r r(=s6                  zParserGenerator.make_labelcCs<t|jj}|jx |D]}||jkr|j|qWdS)N)rrrrr calcfirst)rr/r0r r r rks   zParserGenerator.addfirstsetsc Cs |j|}d|j|<|d}i}i}x|jjD]x\}}||jkr||jkrl|j|}|dkrtd|n|j||j|}|j||||<q0d||<|di||<q0Wi} xJ|jD]>\}} x4| D],} | | krtd|| || | f|| | <qWqW||j|<dS)Nrzrecursion for rule %rrzArule %s is ambiguous; %s is in the first sets of %s as well as %s)rrr%r& ValueErrorr?update) rr0r2r3ZtotalsetZ overlapcheckr4r5fsetZinverseZitsfirstZsymbolr r r r?ss2          zParserGenerator.calcfirstc Csi}d}x|jtjkrx|jtjkr.|jqW|jtj}|jtjd|j\}}|jtj|j ||}t |}|j |t |}|||<|dkr |}q W||fS)N:) typer ENDMARKERNEWLINErexpectr=OP parse_rhsmake_dfar" simplify_dfa) rrrr0azr2ZoldlenZnewlenr r r rs"      zParserGenerator.parsec sfdd}fddt|||g}x|D]}i}x<|jD]2}x,|jD]"\}} |dk rJ| |j|iqJWq>WxRt|jD]B\}} x,|D]} | j| krPqWt| |} |j| |j| |qWq.W|S)Ncsi}|||S)Nr )r3base) addclosurer r closures z)ParserGenerator.make_dfa..closurecs>||kr dSd||<x$|jD]\}}|dkr||qWdS)Nr)r%)r3rNr4r5)rOr r rOs z,ParserGenerator.make_dfa..addclosure)DFAStatenfasetr% setdefaultr$r&r'addarc) rr-finishrPr+r3r%Znfastater4r5rRstr )rOr rJs"        zParserGenerator.make_dfac Cstd||g}xt|D]\}}td|||kr4dp6dx^|jD]T\}}||kr^|j|} nt|} |j||dkrtd| qBtd|| fqBWqWdS)NzDump of NFA forz Statez(final)z -> %dz %s -> %d)print enumerater%r)r"r') rr0r-rUZtodor1r3r4r5jr r r dump_nfas   zParserGenerator.dump_nfacCsltd|x\t|D]P\}}td||jr,dp.dx0t|jjD]\}}td||j|fqBWqWdS)NzDump of DFA forz Statez(final)rWz %s -> %d)rXrYr*r$r%r&r))rr0r2r1r3r4r5r r r dump_dfas  zParserGenerator.dump_dfacCs~d}xt|rxd}xft|D]Z\}}xPt|dt|D]:}||}||kr4||=x|D]}|j||qTWd}Pq4WqWqWdS)NTFr)rYranger" unifystate)rr2Zchangesr1Zstate_irZZstate_jr3r r r rKs zParserGenerator.simplify_dfacCs|j\}}|jdkr||fSt}t}|j||j|x6|jdkrt|j|j\}}|j||j|q@W||fSdS)N|) parse_altr>NFAStaterTr)rrLrMZaaZzzr r r rIs       zParserGenerator.parse_rhscCsP|j\}}x:|jdks*|jtjtjfkrF|j\}}|j||}qW||fS)N([)rbrc) parse_itemr>rDrr=STRINGrT)rrLbr.dr r r r` s    zParserGenerator.parse_altcCs|jdkr>|j|j\}}|jtjd|j|||fS|j\}}|j}|dkr`||fS|j|j||dkr||fS||fSdS)Nrc]+*)rirj)r>rrIrGrrHrT parse_atom)rrLrMr>r r r rds     zParserGenerator.parse_itemcCs|jdkr4|j|j\}}|jtjd||fS|jtjtjfkrpt }t }|j ||j|j||fS|j d|j|jdS)Nrb)z+expected (...) or NAME or STRING, got %s/%s) r>rrIrGrrHrDr=rerarT raise_error)rrLrMr r r rk(s  zParserGenerator.parse_atomcCsD|j|ks|dk r2|j|kr2|jd|||j|j|j}|j|S)Nzexpected %s/%s, got %s/%s)rDr>rmr)rrDr>r r r rG9s zParserGenerator.expectcCsJt|j}x"|dtjtjfkr,t|j}q W|\|_|_|_|_|_ dS)Nr) r5rrCOMMENTNLrDr>Zbeginendline)rtupr r r rAs zParserGenerator.gettokenc Gs^|r8y ||}Wn&dj|gttt|}YnXt||j|jd|jd|jfdS)N rr)joinrmapstr SyntaxErrorrrprq)rmsgargsr r r rmHs  zParserGenerator.raise_error)N)N)rrrrr6r,r(rr?rrJr[r\rKrIr`rdrkrGrrmr r r r r s$  .$  r c@seZdZddZdddZdS)racCs g|_dS)N)r%)rr r r rSszNFAState.__init__NcCs|jj||fdS)N)r%r')rr5r4r r r rTVszNFAState.addarc)N)rrrrrTr r r r raQsrac@s0eZdZddZddZddZddZd Zd S) rQcCs||_||k|_i|_dS)N)rRr*r%)rrRfinalr r r r]s zDFAState.__init__cCs||j|<dS)N)r%)rr5r4r r r rTeszDFAState.addarccCs.x(|jjD]\}}||kr ||j|<q WdS)N)r%r&)roldnewr4r5r r r r^kszDFAState.unifystatecCsX|j|jkrdSt|jt|jkr(dSx*|jjD]\}}||jj|k r4dSq4WdS)NFT)r*r"r%r&get)rotherr4r5r r r __eq__ps zDFAState.__eq__N)rrrrrTr^r__hash__r r r r rQ[s rQ Grammar.txtcCst|}|jS)N)r r6)rpr r r generate_grammarsrN)r) rWrrrZGrammarrobjectr rarQrr r r r sI %__pycache__/pgen.cpython-36.pyc000064400000023071152531664540012351 0ustar003 \5@sdddlmZmZmZGdddejZGdddeZGdddeZGdd d eZ dd d Z d S))grammartokentokenizec@s eZdZdS) PgenGrammarN)__name__ __module__ __qualname__r r */usr/lib64/python3.6/lib2to3/pgen2/pgen.pyrsrc@seZdZd&ddZddZddZdd Zd d Zd d ZddZ ddZ ddZ ddZ ddZ ddZddZddZddZd'd d!Zd"d#Zd$d%ZdS)(ParserGeneratorNcCsld}|dkrt|}|j}||_||_tj|j|_|j|j \|_ |_ |dk rZ|i|_ |j dS)N)openclosefilenamestreamrgenerate_tokensreadline generatorgettokenparsedfas startsymbolfirst addfirstsets)selfrrZ close_streamr r r __init__ szParserGenerator.__init__c Cs*t}t|jj}|j|j|j|jd|jx.|D]&}dt|j }||j |<||j |<qtj|}||j kr@|j |S|jj|df||j |<|SdS)Nr"')r8r9)r"Zlabelsisalphar#Z symbol2labelr'getattrr isinstanceintAssertionErrortok_nametokensevalkeywordsNAMErZopmap)rr.r4r7Zitokenvaluer r r r(=s<                 zParserGenerator.make_labelcCs<t|jj}|jx |D]}||jkr|j|qWdS)N)rrrrr calcfirst)rr/r0r r r rks   zParserGenerator.addfirstsetsc Cs |j|}d|j|<|d}i}i}x|jjD]x\}}||jkr||jkrl|j|}|dkrtd|n|j||j|}|j||||<q0d||<|di||<q0Wi} xJ|jD]>\}} x4| D],} | | krtd|| || | f|| | <qWqW||j|<dS)Nrzrecursion for rule %rrzArule %s is ambiguous; %s is in the first sets of %s as well as %s)rrr%r& ValueErrorrEupdate) rr0r2r3ZtotalsetZ overlapcheckr4r5fsetZinverseZitsfirstZsymbolr r r rEss2          zParserGenerator.calcfirstc Csi}d}x|jtjkrx|jtjkr.|jqW|jtj}|jtjd|j\}}|jtj|j ||}t |}|j |t |}|||<|dkr |}q W||fS)N:) typer ENDMARKERNEWLINErexpectrCOP parse_rhsmake_dfar" simplify_dfa) rrrr0azr2ZoldlenZnewlenr r r rs"      zParserGenerator.parsec st|tstt|tstfdd}fddt|||g}x|D]}i}x<|jD]2}x,|jD]"\}} |dk rf| |j|iqfWqZWxRt|jD]B\}} x,|D]} | j| krPqWt| |} |j | |j | |qWqJW|S)Ncsi}|||S)Nr )r3base) addclosurer r closures z)ParserGenerator.make_dfa..closurecsLt|tst||krdSd||<x$|jD]\}}|dkr*||q*WdS)Nr)r<NFAStater>r%)r3rTr4r5)rUr r rUsz,ParserGenerator.make_dfa..addclosure) r<rWr>DFAStatenfasetr% setdefaultr$r&r'addarc) rr-finishrVr+r3r%Znfastater4r5rYstr )rUr rPs&        zParserGenerator.make_dfac Cstd||g}xt|D]\}}td|||kr4dp6dx^|jD]T\}}||kr^|j|} nt|} |j||dkrtd| qBtd|| fqBWqWdS)NzDump of NFA forz Statez(final)z -> %dz %s -> %d)print enumerater%r)r"r') rr0r-r\Ztodor1r3r4r5jr r r dump_nfas   zParserGenerator.dump_nfacCsltd|x\t|D]P\}}td||jr,dp.dx0t|jjD]\}}td||j|fqBWqWdS)NzDump of DFA forz Statez(final)r^z %s -> %d)r_r`r*r$r%r&r))rr0r2r1r3r4r5r r r dump_dfas  zParserGenerator.dump_dfacCs~d}xt|rxd}xft|D]Z\}}xPt|dt|D]:}||}||kr4||=x|D]}|j||qTWd}Pq4WqWqWdS)NTFr)r`ranger" unifystate)rr2Zchangesr1Zstate_iraZstate_jr3r r r rQs zParserGenerator.simplify_dfacCs|j\}}|jdkr||fSt}t}|j||j|x6|jdkrt|j|j\}}|j||j|q@W||fSdS)N|) parse_altrDrWr[r)rrRrSZaaZzzr r r rOs       zParserGenerator.parse_rhscCsP|j\}}x:|jdks*|jtjtjfkrF|j\}}|j||}qW||fS)N([)rhri) parse_itemrDrJrrCSTRINGr[)rrRbr.dr r r rg s    zParserGenerator.parse_altcCs|jdkr>|j|j\}}|jtjd|j|||fS|j\}}|j}|dkr`||fS|j|j||dkr||fS||fSdS)Nri]+*)rorp)rDrrOrMrrNr[ parse_atom)rrRrSrDr r r rjs     zParserGenerator.parse_itemcCs|jdkr4|j|j\}}|jtjd||fS|jtjtjfkrpt }t }|j ||j|j||fS|j d|j|jdS)Nrh)z+expected (...) or NAME or STRING, got %s/%s) rDrrOrMrrNrJrCrkrWr[ raise_error)rrRrSr r r rq(s  zParserGenerator.parse_atomcCsD|j|ks|dk r2|j|kr2|jd|||j|j|j}|j|S)Nzexpected %s/%s, got %s/%s)rJrDrsr)rrJrDr r r rM9s zParserGenerator.expectcCsJt|j}x"|dtjtjfkr,t|j}q W|\|_|_|_|_|_ dS)Nr) r5rrCOMMENTNLrJrDZbeginendline)rtupr r r rAs zParserGenerator.gettokenc Gs^|r8y ||}Wn&dj|gttt|}YnXt||j|jd|jd|jfdS)N rr)joinrmapstr SyntaxErrorrrvrw)rmsgargsr r r rsHs  zParserGenerator.raise_error)N)N)rrrrr6r,r(rrErrPrbrcrQrOrgrjrqrMrrsr r r r r s$  .$  r c@seZdZddZdddZdS)rWcCs g|_dS)N)r%)rr r r rSszNFAState.__init__NcCs8|dkst|tstt|ts$t|jj||fdS)N)r<r|r>rWr%r')rr5r4r r r r[VszNFAState.addarc)N)rrrrr[r r r r rWQsrWc@s0eZdZddZddZddZddZd Zd S) rXcCsLt|tstttt|ts$tt|ts2t||_||k|_i|_dS)N) r<dictr>r5iterrWrYr*r%)rrYfinalr r r r]s  zDFAState.__init__cCs8t|tst||jkstt|ts*t||j|<dS)N)r<r|r>r%rX)rr5r4r r r r[eszDFAState.addarccCs.x(|jjD]\}}||kr ||j|<q WdS)N)r%r&)roldnewr4r5r r r rekszDFAState.unifystatecCsft|tst|j|jkrdSt|jt|jkr6dSx*|jjD]\}}||jj|k rBdSqBWdS)NFT)r<rXr>r*r"r%r&get)rotherr4r5r r r __eq__ps zDFAState.__eq__N)rrrrr[rer__hash__r r r r rX[s rX Grammar.txtcCst|}|jS)N)r r6)rpr r r generate_grammarsrN)r) r^rrrZGrammarrobjectr rWrXrr r r r sI %__pycache__/driver.cpython-36.opt-2.pyc000064400000007703152531664540013657 0ustar003 \@sdZddgZddlZddlZddlZddlZddlZddlZddlm Z m Z m Z m Z m Z GdddeZdd Zdd dZddZddZddZedkrejee dS)z#Guido van Rossum Driver load_grammarN)grammarparsetokentokenizepgenc@sHeZdZdddZdddZdddZdd d Zdd d Zdd dZdS)rNcCs&||_|dkrtj}||_||_dS)N)rlogging getLoggerloggerconvert)selfrr r r,/usr/lib64/python3.6/lib2to3/pgen2/driver.py__init__ s zDriver.__init__FcCsvtj|j|j}|jd}d}d}}}} } d} x4|D]} | \}}}} } |||fkr|\} }|| kr| d| |7} | }d}||kr| | ||7} |}|tjtjfkr| |7} | \}}|jdr@|d7}d}q@|t j krtj |}|r|j j dt j||| |j||| |fr6|r4|j j dPd} | \}}|jdr@|d7}d}q@Wtjd||| |f|jS)Nrr z%s %r (prefix=%r)zStop.zincomplete input)rZParserrr ZsetuprCOMMENTNLendswithrOPZopmapr debugtok_nameZaddtokenZ ParseErrorZrootnode)rtokensrplinenocolumntypevaluestartendZ line_textprefixZ quintupleZs_linenoZs_columnrrr parse_tokens'sR      zDriver.parse_tokenscCstj|j}|j||S)N)rgenerate_tokensreadliner#)rstreamrrrrrparse_stream_rawWs zDriver.parse_stream_rawcCs |j||S)N)r')rr&rrrr parse_stream\szDriver.parse_streamc Cs*tj|d|}z |j||S|jXdS)Nr)codecsopenr(close)rfilenameencodingrr&rrr parse_file`s zDriver.parse_filecCstjtj|j}|j||S)N)rr$ioStringIOr%r#)rtextrrrrr parse_stringhszDriver.parse_string)NN)F)F)F)NF)F) __name__ __module__ __qualname__rr#r'r(r/r3rrrrrs   0   cCs:tjj|\}}|dkrd}||djtttjdS)Nz.txtr.z.pickle)ospathsplitextjoinmapstrsys version_info)gtheadtailrrr_generate_pickle_namensrC Grammar.txtTFcCs|dkrtj}|dkr t|n|}|s4t|| r|jd|tj|}|r|jd|y|j|Wqtk r}z|jd|WYdd}~XqXnt j }|j ||S)Nz!Generating grammar tables from %szWriting grammar tables to %szWriting failed: %s) r r rC_newerinfor Zgenerate_grammardumpOSErrorrGrammarload)r@Zgpsaveforcer gerrrrus     cCs8tjj|sdStjj|s dStjj|tjj|kS)NFT)r8r9existsgetmtime)abrrrrEs   rEcCsFtjj|rt|Sttjj|}tj||}tj }|j ||S)N) r8r9isfilerrCbasenamepkgutilget_datarrIloads)packageZgrammar_sourceZ pickled_namedatarMrrrload_packaged_grammars   rZcGsF|stjdd}tjtjtjddx|D]}t|dddq,WdS)Nrz %(message)s)levelr&formatT)rKrL)r>argvr Z basicConfigINFOstdoutr)argsr@rrrmains  ra__main__)rDNTFN) __author____all__r*r0r8r rUr>rrrrrr objectrrCrrErZrar4exitintrrrrs"P   __pycache__/grammar.cpython-36.opt-2.pyc000064400000006253152531664540014011 0ustar003 \@stddlZddlZddlmZmZGdddeZddZdZiZ x.ej D]"Z e rJe j \Z Zeeee e <qJWdS) N)tokentokenizec@s<eZdZddZddZddZddZd d Zd d Zd S)GrammarcCs<i|_i|_g|_i|_dg|_i|_i|_i|_d|_dS)NrEMPTY)rr) symbol2number number2symbolstatesdfaslabelskeywordstokens symbol2labelstart)selfr-/usr/lib64/python3.6/lib2to3/pgen2/grammar.py__init__MszGrammar.__init__c Cs2t|d}t|j}tj||dWdQRXdS)Nwb)open_make_deterministic__dict__pickledump)rfilenamefdrrrrXs  z Grammar.dumpc Cs0t|d}tj|}WdQRX|jj|dS)Nrb)rrloadrupdate)rrrrrrrr is z Grammar.loadcCs|jjtj|dS)N)rr!rloads)rZpklrrrr"osz Grammar.loadscCsX|j}x"dD]}t||t||jqW|jdd|_|jdd|_|j|_|S)Nrr r r rr)rr r r rr) __class__setattrgetattrcopyr r r)rnewZ dict_attrrrrr&ssz Grammar.copycCsvddlm}td||jtd||jtd||jtd||jtd||jtd|jdS) Nr)pprintZs2nZn2sr r r r)r(printrr r r r r)rr(rrrreports      zGrammar.reportN) __name__ __module__ __qualname__rrr r"r&r*rrrrrs 6  rcCs^t|tr&tjtdd|jDSt|tr>dd|DSt|trZtdd|DS|S)Ncss|]\}}|t|fVqdS)N)r).0kvrrr sz&_make_deterministic..cSsg|] }t|qSr)r)r.errr sz'_make_deterministic..css|]}t|VqdS)N)r)r.r2rrrr1s) isinstancedict collections OrderedDictsorteditemslisttuple)toprrrrs   ra ( LPAR ) RPAR [ LSQB ] RSQB : COLON , COMMA ; SEMI + PLUS - MINUS * STAR / SLASH | VBAR & AMPER < LESS > GREATER = EQUAL . DOT % PERCENT ` BACKQUOTE { LBRACE } RBRACE @ AT @= ATEQUAL == EQEQUAL != NOTEQUAL <> NOTEQUAL <= LESSEQUAL >= GREATEREQUAL ~ TILDE ^ CIRCUMFLEX << LEFTSHIFT >> RIGHTSHIFT ** DOUBLESTAR += PLUSEQUAL -= MINEQUAL *= STAREQUAL /= SLASHEQUAL %= PERCENTEQUAL &= AMPEREQUAL |= VBAREQUAL ^= CIRCUMFLEXEQUAL <<= LEFTSHIFTEQUAL >>= RIGHTSHIFTEQUAL **= DOUBLESTAREQUAL // DOUBLESLASH //= DOUBLESLASHEQUAL -> RARROW )r6rrrobjectrrZ opmap_rawZopmap splitlineslinesplitopnamer%rrrrsy= __pycache__/conv.cpython-36.opt-2.pyc000064400000007120152531664540013322 0ustar003 \%@s.ddlZddlmZmZGdddejZdS)N)grammartokenc@s,eZdZddZddZddZddZd S) ConvertercCs |j||j||jdS)N)parse_graminit_hparse_graminit_c finish_off)selfZ graminit_hZ graminit_cr */usr/lib64/python3.6/lib2to3/pgen2/conv.pyrun/s  z Converter.runc Csy t|}Wn0tk r<}ztd||fdSd}~XnXi|_i|_d}xn|D]f}|d7}tjd|}| r|jrtd|||jfqT|j\}}t |}||j|<||j|<qTWdS)NzCan't open %s: %sFrz^#define\s+(\w+)\s+(\d+)$z%s(%s): can't parse %sT) openOSErrorprintZ symbol2numberZ number2symbolrematchstripgroupsint) rfilenameferrlinenolinemosymbolnumberr r r r5s&     zConverter.parse_graminit_hc!Csy t|}Wn0tk r<}ztd||fdSd}~XnXd}|dt|}}|dt|}}|dt|}}i}g}x|jdrx|jdrLtjd|}ttt |j \} } } g} xRt | D]F} |dt|}}tjd|}ttt |j \}}| j ||fqW|dt|}}| || | f<|dt|}}qWtjd|}ttt |j \}}g}x^t |D]R} |dt|}}tjd |}ttt |j \} } } || | f} |j | q~W|j ||dt|}}|dt|}}qW||_ i}tjd |}t |jd}xt |D]}|dt|}}tjd |}|jd }ttt |jdd dd\}}}}||}|dt|}}tjd|}i}t|jd}xPt|D]D\}}t|}x0t dD]$}|d|>@rd||d|<qWqW||f||<q4W|dt|}}||_g}|dt|}}tjd|}t |jd}xjt |D]^}|dt|}}tjd|}|j \}}t |}|dkrd}nt|}|j ||fqpW|dt|}}||_|dt|}}|dt|}}tjd|}t |jd}|dt|}}|dt|}}tjd|}t |jd}|dt|}}tjd|}t |jd} | |_|dt|}}y|dt|}}Wntk rYnXdS)NzCan't open %s: %sFrr z static arc z)static arc arcs_(\d+)_(\d+)\[(\d+)\] = {$z\s+{(\d+), (\d+)},$z'static state states_(\d+)\[(\d+)\] = {$z\s+{(\d+), arcs_(\d+)_(\d+)},$zstatic dfa dfas\[(\d+)\] = {$z0\s+{(\d+), "(\w+)", (\d+), (\d+), states_(\d+),$z\s+("(?:\\\d\d\d)*")},$z!static label labels\[(\d+)\] = {$z\s+{(\d+), (0|"\w+")},$0z \s+(\d+),$z\s+{(\d+), labels},$z \s+(\d+)$)r rrnext startswithrrlistmaprrrangeappendstatesgroupeval enumerateorddfaslabelsstart StopIteration)!rrrrrrZallarcsr)rnmkZarcs_ijststater.ZndfasrrxyzfirstZ rawbitsetcZbyter/Znlabelsr0r r r rTs         "        zConverter.parse_graminit_ccCs\i|_i|_xJt|jD]<\}\}}|tjkrB|dk rB||j|<q|dkr||j|<qWdS)N)keywordstokensr,r/rNAME)rZilabeltypevaluer r r rs zConverter.finish_offN)__name__ __module__ __qualname__r rrrr r r r r$s  &r)rZpgen2rrZGrammarrr r r r s__pycache__/parse.cpython-36.opt-2.pyc000064400000006006152531664540013471 0ustar003 \u@s0ddlmZGdddeZGdddeZdS))tokenc@seZdZddZdS) ParseErrorcCs4tj|d||||f||_||_||_||_dS)Nz!%s: type=%r, value=%r, context=%r) Exception__init__msgtypevaluecontext)selfrrrr r +/usr/lib64/python3.6/lib2to3/pgen2/parse.pyrs zParseError.__init__N)__name__ __module__ __qualname__rr r r r rsrc@sHeZdZdddZdddZddZdd Zd d Zd d ZddZ dS)ParserNcCs||_|pdd|_dS)NcSs|S)Nr )grammarnoder r r Wsz!Parser.__init__..)rconvert)r rrr r r r9szParser.__init__cCsH|dkr|jj}|ddgf}|jj|d|f}|g|_d|_t|_dS)N)rstartdfasstackrootnodeset used_names)r rnewnodeZ stackentryr r r setupYs  z Parser.setupcCs:|j|||}x$|jd\}}}|\}} ||} x| D]\} } |jj| \} }|| kr|j||| || }x@||d|fgkr|j|jsdS|jd \}}}|\}} qpWdS| dkr:|jj| }|\}}||kr:|j| |jj| | |Pq:Wd|f| kr$|j|js2td|||qtd|||qWdS) NrrTFztoo much inputz bad inputr) classifyrrZlabelsshiftpoprpushr)r rrr ilabeldfastaterZstatesfirstZarcsinewstatetvZitsdfaZ itsstatesZitsfirstr r r addtokenqs:   zParser.addtokencCsX|tjkr0|jj||jjj|}|dk r0|S|jjj|}|dkrTtd||||S)Nz bad token) rNAMEraddrkeywordsgettokensr)r rrr r$r r r r s  zParser.classifyc CsT|jd\}}}|||df}|j|j|}|dk r@|dj||||f|jd<dS)Nrrrr)rrrappend) r rrr)r r%r&rrr r r r!s  z Parser.shiftc CsB|jd\}}}|d|gf}|||f|jd<|jj|d|fdS)Nrrrr)rr2) r rZnewdfar)r r%r&rrr r r r#s z Parser.pushcCs`|jj\}}}|j|j|}|dk r\|jrL|jd\}}}|dj|n||_|j|j_dS)Nrrr)rr"rrr2rr)r ZpopdfaZpopstateZpopnoderr%r&rr r r r"sz Parser.pop)N)N) r rrrrr,r r!r#r"r r r r rs 0 rN)rrrobjectrr r r r s  __pycache__/tokenize.cpython-36.opt-2.pyc000064400000026167152531664540014221 0ustar003 \NX=@sdZdZddlZddlZddlmZmZddlTddlm Z dd e e Dd d d gZ [ ye Wne k rzeZ YnXd dZddZddZdZdZeedeeeZdZdZdZdZeddZeeeeeZdZeddeeZdeZeeeZed ed!ZeeeeZ d"Z!d#Z"d$Z#d%Z$d&Z%ee%d'e%d(Z&ee%d)e%d*Z'ed+d,d-d.d/d0d1d2d3 Z(d4Z)ed5d6Z*ee(e)e*Z+ee e+e'eZ,ee,Z-ee%d7ed8de%d9ed:dZ.edee&Z/eee/e e+e.eZ0e1e2ej3e-e0e#e$f\Z4Z5Z6Z7ej3e!ej3e"e6e7e6e7e6e7e6e7e6e7e6e7e6e7e6e7e6e7e6e7e6e7e6e7e6e7e6e7e6e7e6e7e6e7e6e7e6e7e6e7e6e7ddddddddd;4Z8iZ9xdD]Z:e:e9e:<qWiZ;xdD]Z:e:e;e:<qWdZGddde=Z?ddZ@e@fdd ZAddZBGdddZCej3dejDZEej3dejDZFddZGddZHdd ZIdd ZJeKdkrddlLZLeMeLjNdkrxeAeOeLjNdjPn eAeLjQjPdS)zKa-Ping Yee z@GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip MontanaroN)BOM_UTF8lookup)*)tokencCsg|]}|ddkr|qS)r_).0xrr./usr/lib64/python3.6/lib2to3/pgen2/tokenize.py %sr tokenizegenerate_tokens untokenizecGsddj|dS)N(|))join)choicesrrr group0srcGs t|dS)Nr)r)rrrr any1srcGs t|dS)N?)r)rrrr maybe2srz[ \f\t]*z #[^\r\n]*z\\\r?\nz [a-zA-Z_]\w*z0[bB]_?[01]+(?:_[01]+)*z(0[xX]_?[\da-fA-F]+(?:_[\da-fA-F]+)*[lL]?z0[oO]?_?[0-7]+(?:_[0-7]+)*[lL]?z[1-9]\d*(?:_\d+)*[lL]?z0[lL]?z[eE][-+]?\d+(?:_\d+)*z\d+(?:_\d+)*\.(?:\d+(?:_\d+)*)?z\.\d+(?:_\d+)*z \d+(?:_\d+)*z\d+(?:_\d+)*[jJ]z[jJ]z[^'\\]*(?:\\.[^'\\]*)*'z[^"\\]*(?:\\.[^"\\]*)*"z%[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''z%[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""z#(?:[uUrRbBfF]|[rR][bB]|[bBuU][rR])?'''"""z'[^\n'\\]*(?:\\.[^\n'\\]*)*'z"[^\n"\\]*(?:\\.[^\n"\\]*)*"z\*\*=?z>>=?z<<=?z<>z!=z//=?z->z[+\-*/%&@|^=<>]=?~z[][(){}]z\r?\nz[:;.,`@]z'[^\n'\\]*(?:\\.[^\n'\\]*)*'z"[^\n"\\]*(?:\\.[^\n"\\]*)*")4rrz'''z"""zr'''zr"""zu'''zu"""zb'''zb"""zf'''zf"""zur'''zur"""zbr'''zbr"""zrb'''zrb"""zR'''zR"""zU'''zU"""zB'''zB"""zF'''zF"""zuR'''zuR"""zUr'''zUr"""zUR'''zUR"""zbR'''zbR"""zBr'''zBr"""zBR'''zBR"""zrB'''zrB"""zRb'''zRb"""zRB'''zRB"""rRuUfFbBr'''r"""R'''R"""u'''u"""U'''U"""b'''b"""B'''B"""f'''f"""F'''F"""ur'''ur"""Ur'''Ur"""uR'''uR"""UR'''UR"""br'''br"""Br'''Br"""bR'''bR"""BR'''BR"""rb'''rb"""Rb'''Rb"""rB'''rB"""RB'''RB"""r'r"R'R"u'u"U'U"b'b"B'B"f'f"F'F"ur'ur"Ur'Ur"uR'uR"UR'UR"br'br"Br'Br"bR'bR"BR'BR"rb'rb"Rb'Rb"rB'rB"RB'RB"c@s eZdZdS) TokenErrorN)__name__ __module__ __qualname__rrrr rwsrwc@s eZdZdS)StopTokenizingN)rxryrzrrrr r{sr{c Cs4|\}}|\}}td||||t|t|fdS)Nz%d,%d-%d,%d: %s %s)printtok_namerepr) typerZxxx_todo_changemeZxxx_todo_changeme1lineZsrowZscolZerowZecolrrr printtokensrc Cs(yt||Wntk r"YnXdS)N) tokenize_loopr{)readline tokeneaterrrr r s cCsxt|D] }||q WdS)N)r)rrZ token_inforrr rsrc@s,eZdZddZddZddZddZd S) UntokenizercCsg|_d|_d|_dS)Nrr)tokensprev_rowprev_col)selfrrr __init__szUntokenizer.__init__cCs*|\}}||j}|r&|jjd|dS)N )rrappend)rstartrowcol col_offsetrrr add_whitespaces zUntokenizer.add_whitespacecCsxv|D]n}t|dkr$|j||P|\}}}}}|j||jj||\|_|_|ttfkr|jd7_d|_qWdj |jS)Nrr) lencompatrrrrrNEWLINENLr)riterablettok_typerrendrrrr rs        zUntokenizer.untokenizec Csd}g}|jj}|\}}|ttfkr,|d7}|ttfkr|t kr|j qBn*|ttfkrd}n|r|r||dd}||qBWdS)NFrTrr) rrNAMENUMBERrrASYNCAWAITINDENTDEDENTpop) rrr startlineindents toks_appendtoknumtokvaltokrrr rs0      zUntokenizer.compatN)rxryrzrrrrrrrr rsrz&^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)s^[ \t\f]*(?:[#\r\n]|$)cCsH|ddjjdd}|dks*|jdr.dS|d ks@|jd rDdS|S)N r-zutf-8zutf-8-latin-1 iso-8859-1 iso-latin-1latin-1- iso-8859-1- iso-latin-1-)rrr)rrr)lowerreplace startswith)orig_encencrrr _get_normal_names rcsdd}d}fdd}fdd}|}|jtrHd|dd}d }|sT|gfS||}|rj||gfStj|s~||gfS|}|s||gfS||}|r|||gfS|||gfS) NFzutf-8c s"yStk rtSXdS)N) StopIterationbytesr)rrr read_or_stop sz%detect_encoding..read_or_stopcsy|jd}Wntk r"dSXtj|}|s6dSt|jd}y t|}Wn tk rptd|YnXr|j dkrtd|d7}|S)Nasciirzunknown encoding: zutf-8zencoding problem: utf-8z-sig) decodeUnicodeDecodeError cookie_rematchrrr LookupError SyntaxErrorname)r line_stringrencodingcodec) bom_foundrr find_cookie&s"   z$detect_encoding..find_cookieTz utf-8-sig)rrblank_rer)rrdefaultrrfirstsecondr)rrr detect_encoding s0         rcCst}|j|S)N)rr)rutrrr rTsc!csd}}}tjdd}}d\}}d}dg} d} d} d} d} xy |}Wntk rdd}YnX|d}dt|}}|rF|std||j|}|r|jd}}t||d||||f||fVd \}}d}nd|r0|d!dd kr0|d"dd kr0t||||t|f|fVd}d}qBn||}||}qBnF|dkrt| rt|s`Pd}xf||kr||d kr|d}n6||d kr|t dt }n||dkrd}nP|d}qfW||krP| r| Vd} ||dkr||dkrh||dj d}|t|}t |||f||t|f|fVt ||d||f|t|f|fVqBt t f||dk||d||f|t|f|fVqB|| d#kr| j |t|d||df||f|fVxt|| d$krJ|| krtdd|||f| dd%} | r.| | d&kr.d} d} d} td||f||f|fVqW| r| r| | d'krd} d} d} n|std|dfd}x||kr8tj||}|r |jd\}}||f||f|}}}|||||}}||ks|dkr|dkrt||||fVq4|dkrft}|dkr8t }n | rBd} | rR| Vd} |||||fVq4|dkr| r| Vd} t ||||fVq4|tkrt|}|j||}|r|jd}|||}| r| Vd} t||||f|fVn||f}||d}|}Pq4|tks@|ddtks@|dd tkr|d(dkr||f}t|pxt|dpxt|d}||dd}}|}Pn | r| Vd} t||||fVq4||kr|d)kr| r|dkrtnt||||fVqt||||f}|dkr| r|} q|dkrx| rx| dtkrx| ddkrxd} | d*} t| d| d| d | dfVd} | r| Vd} |Vnz|dkr| r| Vd} t ||||f|fVd}nF|dkr|d}n|dkr|d}| r| Vd} t||||fVn(t||||f||df|fV|d}qWqBW| rN| Vd} x.| ddD]} td|df|dfdfVq\Wtd|df|dfdfVdS)+Nrr 0123456789rFrzEOF in multi-line stringrz\ rz\ r  z# #z z3unindent does not match any outer indentation levelz zEOF in multi-line statement.T asyncawaitdef\z([{z)]})rr)rrrrrrrr)rrr)stringZ ascii_lettersrrrwrrSTRING ERRORTOKENtabsizerstripCOMMENTrrrIndentationErrorr pseudoprogspanrr triple_quotedendprogs single_quotedrrrOP ENDMARKER)!rlnumparenlev continuedZ namecharsnumcharscontstrneedcontcontlinerstashed async_defasync_def_indent async_def_nlrposmaxstrstartendprogendmatchrcolumn comment_tokennl_pos pseudomatchrsposeposrinitialnewlinerindentrrr risp     *                             __main__)*rrr&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrM)*rrrNrOrPrQrRrSrTrUrVrWrXrYrZr[r\r]r^r_r`rarbrcrdrerfrgrhrirjrkrlrmrnrorprqrrrsrtru)R __author__ __credits__rrecodecsrrZlib2to3.pgen2.tokenrrdir__all__r NameErrorstrrrr WhitespaceCommentIgnoreName Binnumber Hexnumber Octnumber Decnumber IntnumberExponent PointfloatExpfloat Floatnumber ImagnumberNumberSingleDoubleSingle3Double3Z _litprefixTripleStringOperatorBracketSpecialFunny PlainTokenTokenContStr PseudoExtras PseudoTokenlistmapcompileZ tokenprogrZ single3progZ double3progrrrrr Exceptionrwr{rr rrASCIIrrrrrrrxsysrargvopenrstdinrrrr s              8 Ic __pycache__/tokenize.cpython-36.opt-1.pyc000064400000035722152531664540014215 0ustar003 \NX=@sdZdZdZddlZddlZddlmZmZddlTddl m Z d d e e Dd d d gZ [ ye Wnek r~eZ YnXddZddZddZdZdZeedeeeZdZdZdZdZeddZeeeeeZdZeddeeZd eZeeeZed!ed"Z ee eeZ!d#Z"d$Z#d%Z$d&Z%d'Z&ee&d(e&d)Z'ee&d*e&d+Z(ed,d-d.d/d0d1d2d3d4 Z)d5Z*ed6d7Z+ee)e*e+Z,ee!e,e(eZ-ee-Z.ee&d8ed9de&d:ed;dZ/edee'Z0eee0e!e,e/eZ1e2e3ej4e.e1e$e%f\Z5Z6Z7Z8ej4e"ej4e#e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8ddddddddd<4Z9iZ:xdD]Z;e;e:e;<qWiZZ?Gddde>Z@ddZAeAfdd ZBddZCGdddZDej4dejEZFej4dejEZGddZHddZIdd ZJdd ZKeLdkrddlMZMeNeMjOdkr|eBePeMjOdjQn eBeMjRjQdS)aTokenization help for Python programs. generate_tokens(readline) is a generator that breaks a stream of text into Python tokens. It accepts a readline-like method which is called repeatedly to get the next line of input (or "" for EOF). It generates 5-tuples with these members: the token type (see token.py) the token (a string) the starting (row, column) indices of the token (a 2-tuple of ints) the ending (row, column) indices of the token (a 2-tuple of ints) the original line (string) It is designed to match the working of the Python tokenizer exactly, except that it produces COMMENT tokens for comments and gives type OP for all operators Older entry points tokenize_loop(readline, tokeneater) tokenize(readline, tokeneater=printtoken) are the same, except instead of generating tokens, tokeneater is a callback function to which the 5 fields described above are passed as 5 arguments, each time a new token is found.zKa-Ping Yee z@GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip MontanaroN)BOM_UTF8lookup)*)tokencCsg|]}|ddkr|qS)r_).0xrr./usr/lib64/python3.6/lib2to3/pgen2/tokenize.py %sr tokenizegenerate_tokens untokenizecGsddj|dS)N(|))join)choicesrrr group0srcGs t|dS)Nr)r)rrrr any1srcGs t|dS)N?)r)rrrr maybe2srz[ \f\t]*z #[^\r\n]*z\\\r?\nz [a-zA-Z_]\w*z0[bB]_?[01]+(?:_[01]+)*z(0[xX]_?[\da-fA-F]+(?:_[\da-fA-F]+)*[lL]?z0[oO]?_?[0-7]+(?:_[0-7]+)*[lL]?z[1-9]\d*(?:_\d+)*[lL]?z0[lL]?z[eE][-+]?\d+(?:_\d+)*z\d+(?:_\d+)*\.(?:\d+(?:_\d+)*)?z\.\d+(?:_\d+)*z \d+(?:_\d+)*z\d+(?:_\d+)*[jJ]z[jJ]z[^'\\]*(?:\\.[^'\\]*)*'z[^"\\]*(?:\\.[^"\\]*)*"z%[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''z%[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""z#(?:[uUrRbBfF]|[rR][bB]|[bBuU][rR])?'''"""z'[^\n'\\]*(?:\\.[^\n'\\]*)*'z"[^\n"\\]*(?:\\.[^\n"\\]*)*"z\*\*=?z>>=?z<<=?z<>z!=z//=?z->z[+\-*/%&@|^=<>]=?~z[][(){}]z\r?\nz[:;.,`@]z'[^\n'\\]*(?:\\.[^\n'\\]*)*'z"[^\n"\\]*(?:\\.[^\n"\\]*)*")4rrz'''z"""zr'''zr"""zu'''zu"""zb'''zb"""zf'''zf"""zur'''zur"""zbr'''zbr"""zrb'''zrb"""zR'''zR"""zU'''zU"""zB'''zB"""zF'''zF"""zuR'''zuR"""zUr'''zUr"""zUR'''zUR"""zbR'''zbR"""zBr'''zBr"""zBR'''zBR"""zrB'''zrB"""zRb'''zRb"""zRB'''zRB"""rRuUfFbBr'''r"""R'''R"""u'''u"""U'''U"""b'''b"""B'''B"""f'''f"""F'''F"""ur'''ur"""Ur'''Ur"""uR'''uR"""UR'''UR"""br'''br"""Br'''Br"""bR'''bR"""BR'''BR"""rb'''rb"""Rb'''Rb"""rB'''rB"""RB'''RB"""r'r"R'R"u'u"U'U"b'b"B'B"f'f"F'F"ur'ur"Ur'Ur"uR'uR"UR'UR"br'br"Br'Br"bR'bR"BR'BR"rb'rb"Rb'Rb"rB'rB"RB'RB"c@s eZdZdS) TokenErrorN)__name__ __module__ __qualname__rrrr rwsrwc@s eZdZdS)StopTokenizingN)rxryrzrrrr r{sr{c Cs4|\}}|\}}td||||t|t|fdS)Nz%d,%d-%d,%d: %s %s)printtok_namerepr) typerZxxx_todo_changemeZxxx_todo_changeme1lineZsrowZscolZerowZecolrrr printtokensrc Cs(yt||Wntk r"YnXdS)a: The tokenize() function accepts two parameters: one representing the input stream, and one providing an output mechanism for tokenize(). The first parameter, readline, must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string. The second parameter, tokeneater, must also be a callable object. It is called once for each token, with five arguments, corresponding to the tuples generated by generate_tokens(). N) tokenize_loopr{)readline tokeneaterrrr r s cCsxt|D] }||q WdS)N)r)rrZ token_inforrr rsrc@s,eZdZddZddZddZddZd S) UntokenizercCsg|_d|_d|_dS)Nrr)tokensprev_rowprev_col)selfrrr __init__szUntokenizer.__init__cCs*|\}}||j}|r&|jjd|dS)N )rrappend)rstartrowcol col_offsetrrr add_whitespaces zUntokenizer.add_whitespacecCsxv|D]n}t|dkr$|j||P|\}}}}}|j||jj||\|_|_|ttfkr|jd7_d|_qWdj |jS)Nrr) lencompatrrrrrNEWLINENLr)riterablettok_typerrendrrrr rs        zUntokenizer.untokenizec Csd}g}|jj}|\}}|ttfkr,|d7}|ttfkr|t kr|j qBn*|ttfkrd}n|r|r||dd}||qBWdS)NFrTrr) rrNAMENUMBERrrASYNCAWAITINDENTDEDENTpop) rrr startlineindents toks_appendtoknumtokvaltokrrr rs0      zUntokenizer.compatN)rxryrzrrrrrrrr rsrz&^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)s^[ \t\f]*(?:[#\r\n]|$)cCsH|ddjjdd}|dks*|jdr.dS|d ks@|jdrDdS|S)z(Imitates get_normal_name in tokenizer.c.N r-zutf-8zutf-8-latin-1 iso-8859-1 iso-latin-1latin-1- iso-8859-1- iso-latin-1-)rrr)rrr)lowerreplace startswith)orig_encencrrr _get_normal_names rcsdd}d}fdd}fdd}|}|jtrHd|d d}d }|sT|gfS||}|rj||gfStj|s~||gfS|}|s||gfS||}|r|||gfS|||gfS) a The detect_encoding() function is used to detect the encoding that should be used to decode a Python source file. It requires one argument, readline, in the same way as the tokenize() generator. It will call readline a maximum of twice, and return the encoding used (as a string) and a list of any lines (left as bytes) it has read in. It detects the encoding from the presence of a utf-8 bom or an encoding cookie as specified in pep-0263. If both a bom and a cookie are present, but disagree, a SyntaxError will be raised. If the encoding cookie is an invalid charset, raise a SyntaxError. Note that if a utf-8 bom is found, 'utf-8-sig' is returned. If no encoding is specified, then the default of 'utf-8' will be returned. FNzutf-8c s"yStk rtSXdS)N) StopIterationbytesr)rrr read_or_stop sz%detect_encoding..read_or_stopcsy|jd}Wntk r"dSXtj|}|s6dSt|jd}y t|}Wn tk rptd|YnXr|j dkrtd|d7}|S)Nasciirzunknown encoding: zutf-8zencoding problem: utf-8z-sig) decodeUnicodeDecodeError cookie_rematchrrr LookupError SyntaxErrorname)r line_stringrencodingcodec) bom_foundrr find_cookie&s"   z$detect_encoding..find_cookieTz utf-8-sig)rrblank_rer)rrdefaultrrfirstsecondr)rrr detect_encoding s0         rcCst}|j|S)aTransform tokens back into Python source code. Each element returned by the iterable must be a token sequence with at least two elements, a token number and token value. If only two tokens are passed, the resulting output is poor. Round-trip invariant for full input: Untokenized source will match input source exactly Round-trip invariant for limited intput: # Output text will tokenize the back to the input t1 = [tok[:2] for tok in generate_tokens(f.readline)] newcode = untokenize(t1) readline = iter(newcode.splitlines(1)).next t2 = [tok[:2] for tokin generate_tokens(readline)] assert t1 == t2 )rr)rutrrr rTsc!csd}}}tjdd}}d \}}d}dg} d} d} d} d} xy |}Wntk rdd}YnX|d}dt|}}|rF|std||j|}|r|jd}}t||d||||f||fVd!\}}d}nd|r0|d"dd kr0|d#dd kr0t||||t|f|fVd}d}qBn||}||}qBnF|dkrt| rt|s`Pd}xf||kr||d kr|d}n6||dkr|t dt }n||dkrd}nP|d}qfW||krP| r| Vd} ||dkr||dkrh||dj d}|t|}t |||f||t|f|fVt ||d||f|t|f|fVqBt t f||dk||d||f|t|f|fVqB|| d$kr| j |t|d||df||f|fVxt|| d%krJ|| krtdd|||f| dd&} | r.| | d'kr.d} d} d} td||f||f|fVqW| r| r| | d(krd} d} d} n|std|dfd}x||kr8tj||}|r |jd\}}||f||f|}}}|||||}}||ks|dkr|dkrt||||fVq4|dkrft}|dkr8t }n | rBd} | rR| Vd} |||||fVq4|dkr| r| Vd} t ||||fVq4|tkrt|}|j||}|r|jd}|||}| r| Vd} t||||f|fVn||f}||d}|}Pq4|tks@|dd tks@|dd tkr|d)dkr||f}t|pxt|dpxt|d }||dd}}|}Pn | r| Vd} t||||fVq4||kr|d*kr| r|dkrtnt||||fVqt||||f}|dkr| r|} q|dkrx| rx| dtkrx| ddkrxd} | d+} t| d| d | d | dfVd} | r| Vd} |Vnz|dkr| r| Vd} t ||||f|fVd}nF|dkr|d}n|dkr|d}| r| Vd} t||||fVn(t||||f||df|fV|d}qWqBW| rN| Vd} x.| ddD]} td|df|dfdfVq\Wtd|df|dfdfVdS),aT The generate_tokens() generator requires one argument, readline, which must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string. Alternately, readline can be a callable function terminating with StopIteration: readline = open(myfile).next # Example of alternate readline The generator produces 5-tuples with these members: the token type; the token string; a 2-tuple (srow, scol) of ints specifying the row and column where the token begins in the source; a 2-tuple (erow, ecol) of ints specifying the row and column where the token ends in the source; and the line on which the token was found. The line passed is the logical line; continuation lines are included. rr 0123456789rNFrzEOF in multi-line stringrz\ rz\ r  z# #z z3unindent does not match any outer indentation levelz zEOF in multi-line statement.T asyncawaitdef\z([{z)]})rr)rrrrrrrr)rrr)stringZ ascii_lettersrrrwrrSTRING ERRORTOKENtabsizerstripCOMMENTrrrIndentationErrorr pseudoprogspanrr triple_quotedendprogs single_quotedrrrOP ENDMARKER)!rlnumparenlev continuedZ namecharsnumcharscontstrneedcontcontlinerstashed async_defasync_def_indent async_def_nlrposmaxstrstartendprogendmatchrcolumn comment_tokennl_pos pseudomatchrsposeposrinitialnewlinerindentrrr risp     *                             __main__)*rrr&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrM)*rrrNrOrPrQrRrSrTrUrVrWrXrYrZr[r\r]r^r_r`rarbrcrdrerfrgrhrirjrkrlrmrnrorprqrrrsrtru)S__doc__ __author__ __credits__rrecodecsrrZlib2to3.pgen2.tokenrrdir__all__r NameErrorstrrrr WhitespaceCommentIgnoreName Binnumber Hexnumber Octnumber Decnumber IntnumberExponent PointfloatExpfloat Floatnumber ImagnumberNumberSingleDoubleSingle3Double3Z _litprefixTripleStringOperatorBracketSpecialFunny PlainTokenTokenContStr PseudoExtras PseudoTokenlistmapcompileZ tokenprogrZ single3progZ double3progrrrrr Exceptionrwr{rr rrASCIIrrrrrrrxsysrargvopenrstdinrrrr s              8 Ic __pycache__/token.cpython-36.opt-2.pyc000064400000003412152531664540013475 0ustar003 \@sLdZdZdZdZdZdZdZdZdZd Z d Z d Z d Z d Z dZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZd Z d!Z!d"Z"d#Z#d$Z$d%Z%d&Z&d'Z'd(Z(d)Z)d*Z*d+Z+d,Z,d-Z-d.Z.d/Z/d0Z0d1Z1d2Z2d3Z3d4Z4d5Z5d6Z6d7Z7d8Z8d9Z9d:Z:d;Z;de?j@D]$\ZAZBeCeBeCdkreAe=eB<qWd=d>ZDd?d@ZEdAdBZFdCS)D  !"#$%&'()*+,-./0123456789:;cCs|tkS)N) NT_OFFSET)xr@+/usr/lib64/python3.6/lib2to3/pgen2/token.py ISTERMINALNsrBcCs|tkS)N)r>)r?r@r@rA ISNONTERMINALQsrCcCs|tkS)N) ENDMARKER)r?r@r@rAISEOFTsrEN)GrDNAMENUMBERSTRINGNEWLINEINDENTDEDENTLPARRPARLSQBRSQBCOLONCOMMASEMIPLUSMINUSSTARSLASHVBARAMPERLESSGREATEREQUALDOTPERCENTZ BACKQUOTELBRACERBRACEEQEQUALNOTEQUAL LESSEQUAL GREATEREQUALTILDE CIRCUMFLEX LEFTSHIFT RIGHTSHIFT DOUBLESTAR PLUSEQUALMINEQUAL STAREQUAL SLASHEQUAL PERCENTEQUAL AMPEREQUAL VBAREQUALCIRCUMFLEXEQUALLEFTSHIFTEQUALRIGHTSHIFTEQUALDOUBLESTAREQUAL DOUBLESLASHDOUBLESLASHEQUALATATEQUALOPCOMMENTNLRARROWAWAITASYNC ERRORTOKENN_TOKENSr>tok_namelistglobalsitems_nameZ_valuetyperBrCrEr@r@r@rA s__pycache__/token.cpython-36.pyc000064400000003474152531664540012545 0ustar003 \@sPdZdZdZdZdZdZdZdZdZd Z d Z d Z d Z d Z dZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZd Z d!Z!d"Z"d#Z#d$Z$d%Z%d&Z&d'Z'd(Z(d)Z)d*Z*d+Z+d,Z,d-Z-d.Z.d/Z/d0Z0d1Z1d2Z2d3Z3d4Z4d5Z5d6Z6d7Z7d8Z8d9Z9d:Z:d;Z;dx6e?e@jAD]$\ZBZCeDeCeDdkr eBe>eC<q Wd>d?ZEd@dAZFdBdCZGdDS)Ez!Token constants (from "token.h").  !"#$%&'()*+,-./0123456789:;cCs|tkS)N) NT_OFFSET)xr@+/usr/lib64/python3.6/lib2to3/pgen2/token.py ISTERMINALNsrBcCs|tkS)N)r>)r?r@r@rA ISNONTERMINALQsrCcCs|tkS)N) ENDMARKER)r?r@r@rAISEOFTsrEN)H__doc__rDNAMENUMBERSTRINGNEWLINEINDENTDEDENTLPARRPARLSQBRSQBCOLONCOMMASEMIPLUSMINUSSTARSLASHVBARAMPERLESSGREATEREQUALDOTPERCENTZ BACKQUOTELBRACERBRACEEQEQUALNOTEQUAL LESSEQUAL GREATEREQUALTILDE CIRCUMFLEX LEFTSHIFT RIGHTSHIFT DOUBLESTAR PLUSEQUALMINEQUAL STAREQUAL SLASHEQUAL PERCENTEQUAL AMPEREQUAL VBAREQUALCIRCUMFLEXEQUALLEFTSHIFTEQUALRIGHTSHIFTEQUALDOUBLESTAREQUAL DOUBLESLASHDOUBLESLASHEQUALATATEQUALOPCOMMENTNLRARROWAWAITASYNC ERRORTOKENN_TOKENSr>tok_namelistglobalsitems_nameZ_valuetyperBrCrEr@r@r@rAs__pycache__/parse.cpython-36.opt-1.pyc000064400000014233152531664540013471 0ustar003 \u@s4dZddlmZGdddeZGdddeZdS)zParser engine for the grammar tables generated by pgen. The grammar table must be loaded first. See Parser/parser.c in the Python distribution for additional info on how this parsing engine works. )tokenc@seZdZdZddZdS) ParseErrorz(Exception to signal the parser is stuck.cCs4tj|d||||f||_||_||_||_dS)Nz!%s: type=%r, value=%r, context=%r) Exception__init__msgtypevaluecontext)selfrrrr r +/usr/lib64/python3.6/lib2to3/pgen2/parse.pyrs zParseError.__init__N)__name__ __module__ __qualname____doc__rr r r r rsrc@sLeZdZdZdddZdddZddZd d Zd d Zd dZ ddZ dS)Parsera5Parser engine. The proper usage sequence is: p = Parser(grammar, [converter]) # create instance p.setup([start]) # prepare for parsing : if p.addtoken(...): # parse a token; may raise ParseError break root = p.rootnode # root of abstract syntax tree A Parser instance may be reused by calling setup() repeatedly. A Parser instance contains state pertaining to the current token sequence, and should not be used concurrently by different threads to parse separate token sequences. See driver.py for how to get input tokens by tokenizing a file or string. Parsing is complete when addtoken() returns True; the root of the abstract syntax tree can then be retrieved from the rootnode instance variable. When a syntax error occurs, addtoken() raises the ParseError exception. There is no error recovery; the parser cannot be used after a syntax error was reported (but it can be reinitialized by calling setup()). NcCs||_|pdd|_dS)aConstructor. The grammar argument is a grammar.Grammar instance; see the grammar module for more information. The parser is not ready yet for parsing; you must call the setup() method to get it started. The optional convert argument is a function mapping concrete syntax tree nodes to abstract syntax tree nodes. If not given, no conversion is done and the syntax tree produced is the concrete syntax tree. If given, it must be a function of two arguments, the first being the grammar (a grammar.Grammar instance), and the second being the concrete syntax tree node to be converted. The syntax tree is converted from the bottom up. A concrete syntax tree node is a (type, value, context, nodes) tuple, where type is the node type (a token or symbol number), value is None for symbols and a string for tokens, context is None or an opaque value used for error reporting (typically a (lineno, offset) pair), and nodes is a list of children for symbols, and None for tokens. An abstract syntax tree node may be anything; this is entirely up to the converter function. cSs|S)Nr )grammarnoder r r Wsz!Parser.__init__..N)rconvert)r rrr r r r9szParser.__init__cCsH|dkr|jj}|ddgf}|jj|d|f}|g|_d|_t|_dS)aPrepare for parsing. This *must* be called before starting to parse. The optional argument is an alternative start symbol; it defaults to the grammar's start symbol. You can use a Parser instance to parse any number of programs; each time you call setup() the parser is reset to an initial state determined by the (implicit or explicit) start symbol. N)rstartdfasstackrootnodeset used_names)r rnewnodeZ stackentryr r r setupYs  z Parser.setupcCs:|j|||}x$|jd \}}}|\}} ||} x| D]\} } |jj| \} }|| kr|j||| || }x@||d|fgkr|j|jsdS|jd \}}}|\}} qpWdS| dkr:|jj| }|\}}||kr:|j| |jj| | |Pq:Wd|f| kr$|j|js2td|||qtd|||qWdS) z s  __pycache__/__init__.cpython-36.opt-2.pyc000064400000000203152531664540014107 0ustar003 \@sdS)Nrrr./usr/lib64/python3.6/lib2to3/pgen2/__init__.pys__pycache__/literals.cpython-36.pyc000064400000002776152531664540013250 0ustar003 \O @sPdZddlZddddddd d d d d ZddZddZddZedkrLedS)ztdD]2}t|}t|}t|}||kr t||||q WdS)N)ranger r*r0print)r%cr.er&r&r'test2s r6__main__)__doc__r,rr(r0r6__name__r&r&r&r's   __pycache__/pgen.cpython-36.opt-2.pyc000064400000022165152531664540013314 0ustar003 \5@sdddlmZmZmZGdddejZGdddeZGdddeZGdd d eZ dd d Z d S))grammartokentokenizec@s eZdZdS) PgenGrammarN)__name__ __module__ __qualname__r r */usr/lib64/python3.6/lib2to3/pgen2/pgen.pyrsrc@seZdZd&ddZddZddZdd Zd d Zd d ZddZ ddZ ddZ ddZ ddZ ddZddZddZddZd'd d!Zd"d#Zd$d%ZdS)(ParserGeneratorNcCsld}|dkrt|}|j}||_||_tj|j|_|j|j \|_ |_ |dk rZ|i|_ |j dS)N)openclosefilenamestreamrgenerate_tokensreadline generatorgettokenparsedfas startsymbolfirst addfirstsets)selfrrZ close_streamr r r __init__ szParserGenerator.__init__c Cs*t}t|jj}|j|j|j|jd|jx.|D]&}dt|j }||j |<||j |<qtt|d}||jkrz|j|S|jj|df||j|<|Snt |}|djr||j kr|j |S|jjtj |f||j |<|Sn>t j |}||jkr|j|S|jj|df||j|<|SdS)Nr)r"Zlabelsisalphar#Z symbol2labelr'getattrrtokensevalkeywordsNAMErZopmap)rr.r4r7Zitokenvaluer r r r(=s6                  zParserGenerator.make_labelcCs<t|jj}|jx |D]}||jkr|j|qWdS)N)rrrrr calcfirst)rr/r0r r r rks   zParserGenerator.addfirstsetsc Cs |j|}d|j|<|d}i}i}x|jjD]x\}}||jkr||jkrl|j|}|dkrtd|n|j||j|}|j||||<q0d||<|di||<q0Wi} xJ|jD]>\}} x4| D],} | | krtd|| || | f|| | <qWqW||j|<dS)Nrzrecursion for rule %rrzArule %s is ambiguous; %s is in the first sets of %s as well as %s)rrr%r& ValueErrorr?update) rr0r2r3ZtotalsetZ overlapcheckr4r5fsetZinverseZitsfirstZsymbolr r r r?ss2          zParserGenerator.calcfirstc Csi}d}x|jtjkrx|jtjkr.|jqW|jtj}|jtjd|j\}}|jtj|j ||}t |}|j |t |}|||<|dkr |}q W||fS)N:) typer ENDMARKERNEWLINErexpectr=OP parse_rhsmake_dfar" simplify_dfa) rrrr0azr2ZoldlenZnewlenr r r rs"      zParserGenerator.parsec sfdd}fddt|||g}x|D]}i}x<|jD]2}x,|jD]"\}} |dk rJ| |j|iqJWq>WxRt|jD]B\}} x,|D]} | j| krPqWt| |} |j| |j| |qWq.W|S)Ncsi}|||S)Nr )r3base) addclosurer r closures z)ParserGenerator.make_dfa..closurecs>||kr dSd||<x$|jD]\}}|dkr||qWdS)Nr)r%)r3rNr4r5)rOr r rOs z,ParserGenerator.make_dfa..addclosure)DFAStatenfasetr% setdefaultr$r&r'addarc) rr-finishrPr+r3r%Znfastater4r5rRstr )rOr rJs"        zParserGenerator.make_dfac Cstd||g}xt|D]\}}td|||kr4dp6dx^|jD]T\}}||kr^|j|} nt|} |j||dkrtd| qBtd|| fqBWqWdS)NzDump of NFA forz Statez(final)z -> %dz %s -> %d)print enumerater%r)r"r') rr0r-rUZtodor1r3r4r5jr r r dump_nfas   zParserGenerator.dump_nfacCsltd|x\t|D]P\}}td||jr,dp.dx0t|jjD]\}}td||j|fqBWqWdS)NzDump of DFA forz Statez(final)rWz %s -> %d)rXrYr*r$r%r&r))rr0r2r1r3r4r5r r r dump_dfas  zParserGenerator.dump_dfacCs~d}xt|rxd}xft|D]Z\}}xPt|dt|D]:}||}||kr4||=x|D]}|j||qTWd}Pq4WqWqWdS)NTFr)rYranger" unifystate)rr2Zchangesr1Zstate_irZZstate_jr3r r r rKs zParserGenerator.simplify_dfacCs|j\}}|jdkr||fSt}t}|j||j|x6|jdkrt|j|j\}}|j||j|q@W||fSdS)N|) parse_altr>NFAStaterTr)rrLrMZaaZzzr r r rIs       zParserGenerator.parse_rhscCsP|j\}}x:|jdks*|jtjtjfkrF|j\}}|j||}qW||fS)N([)rbrc) parse_itemr>rDrr=STRINGrT)rrLbr.dr r r r` s    zParserGenerator.parse_altcCs|jdkr>|j|j\}}|jtjd|j|||fS|j\}}|j}|dkr`||fS|j|j||dkr||fS||fSdS)Nrc]+*)rirj)r>rrIrGrrHrT parse_atom)rrLrMr>r r r rds     zParserGenerator.parse_itemcCs|jdkr4|j|j\}}|jtjd||fS|jtjtjfkrpt }t }|j ||j|j||fS|j d|j|jdS)Nrb)z+expected (...) or NAME or STRING, got %s/%s) r>rrIrGrrHrDr=rerarT raise_error)rrLrMr r r rk(s  zParserGenerator.parse_atomcCsD|j|ks|dk r2|j|kr2|jd|||j|j|j}|j|S)Nzexpected %s/%s, got %s/%s)rDr>rmr)rrDr>r r r rG9s zParserGenerator.expectcCsJt|j}x"|dtjtjfkr,t|j}q W|\|_|_|_|_|_ dS)Nr) r5rrCOMMENTNLrDr>Zbeginendline)rtupr r r rAs zParserGenerator.gettokenc Gs^|r8y ||}Wn&dj|gttt|}YnXt||j|jd|jd|jfdS)N rr)joinrmapstr SyntaxErrorrrprq)rmsgargsr r r rmHs  zParserGenerator.raise_error)N)N)rrrrr6r,r(rr?rrJr[r\rKrIr`rdrkrGrrmr r r r r s$  .$  r c@seZdZddZdddZdS)racCs g|_dS)N)r%)rr r r rSszNFAState.__init__NcCs|jj||fdS)N)r%r')rr5r4r r r rTVszNFAState.addarc)N)rrrrrTr r r r raQsrac@s0eZdZddZddZddZddZd Zd S) rQcCs||_||k|_i|_dS)N)rRr*r%)rrRfinalr r r r]s zDFAState.__init__cCs||j|<dS)N)r%)rr5r4r r r rTeszDFAState.addarccCs.x(|jjD]\}}||kr ||j|<q WdS)N)r%r&)roldnewr4r5r r r r^kszDFAState.unifystatecCsX|j|jkrdSt|jt|jkr(dSx*|jjD]\}}||jj|k r4dSq4WdS)NFT)r*r"r%r&get)rotherr4r5r r r __eq__ps zDFAState.__eq__N)rrrrrTr^r__hash__r r r r rQ[s rQ Grammar.txtcCst|}|jS)N)r r6)rpr r r generate_grammarsrN)r) rWrrrZGrammarrobjectr rarQrr r r r sI %__pycache__/conv.cpython-36.opt-1.pyc000064400000014004152531664540013320 0ustar003 \%@s2dZddlZddlmZmZGdddejZdS)aConvert graminit.[ch] spit out by pgen to Python code. Pgen is the Python parser generator. It is useful to quickly create a parser from a grammar file in Python's grammar notation. But I don't want my parsers to be written in C (yet), so I'm translating the parsing tables to Python data structures and writing a Python parse engine. Note that the token numbers are constants determined by the standard Python tokenizer. The standard token module defines these numbers and their names (the names are not used much). The token numbers are hardcoded into the Python tokenizer and into pgen. A Python implementation of the Python tokenizer is also available, in the standard tokenize module. On the other hand, symbol numbers (representing the grammar's non-terminals) are assigned by pgen based on the actual grammar input. Note: this module is pretty much obsolete; the pgen module generates equivalent grammar tables directly from the Grammar.txt input file without having to invoke the Python pgen C program. N)grammartokenc@s0eZdZdZddZddZddZdd Zd S) Convertera2Grammar subclass that reads classic pgen output files. The run() method reads the tables as produced by the pgen parser generator, typically contained in two C files, graminit.h and graminit.c. The other methods are for internal use only. See the base class for more documentation. cCs |j||j||jdS)z@rd||d|<qWqW||f||<q4W|dt|}}||_g}|dt|}}tjd|}t |jd}xjt |D]^}|dt|}}tjd|}|j \}}t |}|dkrd}nt|}|j ||fqpW|dt|}}||_|dt|}}|dt|}}tjd|}t |jd}|dt|}}|dt|}}tjd|}t |jd}|dt|}}tjd|}t |jd} | |_|dt|}}y|dt|}}Wntk rYnXdS)aParse the .c file written by pgen. (Internal) The file looks as follows. The first two lines are always this: #include "pgenheaders.h" #include "grammar.h" After that come four blocks: 1) one or more state definitions 2) a table defining dfas 3) a table defining labels 4) a struct defining the grammar A state definition has the following form: - one or more arc arrays, each of the form: static arc arcs__[] = { {, }, ... }; - followed by a state array, of the form: static state states_[] = { {, arcs__}, ... }; zCan't open %s: %sFNrr z static arc z)static arc arcs_(\d+)_(\d+)\[(\d+)\] = {$z\s+{(\d+), (\d+)},$z'static state states_(\d+)\[(\d+)\] = {$z\s+{(\d+), arcs_(\d+)_(\d+)},$zstatic dfa dfas\[(\d+)\] = {$z0\s+{(\d+), "(\w+)", (\d+), (\d+), states_(\d+),$z\s+("(?:\\\d\d\d)*")},$z!static label labels\[(\d+)\] = {$z\s+{(\d+), (0|"\w+")},$0z \s+(\d+),$z\s+{(\d+), labels},$z \s+(\d+)$)r rrnext startswithrrlistmaprrrangeappendstatesgroupeval enumerateorddfaslabelsstart StopIteration)!rrrrrrZallarcsr)rnmkZarcs_ijststater.ZndfasrrxyzfirstZ rawbitsetcZbyter/Znlabelsr0r r r rTs         "        zConverter.parse_graminit_ccCs\i|_i|_xJt|jD]<\}\}}|tjkrB|dk rB||j|<q|dkr||j|<qWdS)z1Create additional useful structures. (Internal).N)keywordstokensr,r/rNAME)rZilabeltypevaluer r r rs zConverter.finish_offN)__name__ __module__ __qualname____doc__r rrrr r r r r$s  &r)rHrZpgen2rrZGrammarrr r r r s__pycache__/literals.cpython-36.opt-2.pyc000064400000002365152531664540014202 0ustar003 \O @sLddlZdddddddd d d d Zd dZddZddZedkrHedS)N     '"\) abfnrtvr r r c Cs|jdd\}}tj|}|dk r&|S|jdr|dd}t|dkrTtd|yt|d}Wqtk rtd|YqXn0yt|d}Wn tk rtd|YnXt|S) Nrxz!invalid hex string escape ('\%s')z#invalid octal string escape ('\%s'))groupsimple_escapesget startswithlen ValueErrorintchr)malltailescZhexesir%./usr/lib64/python3.6/lib2to3/pgen2/literals.pyescapes"     r'cCsH|d}|dd|dkr$|d}|t|t| }tjdt|S)Nrz)\\(\'|\"|\\|[abfnrtv]|x.{0,2}|[0-7]{1,3}))rresubr')sqr%r%r& evalString(s r-cCsDx>tdD]2}t|}t|}t|}||kr t||||q WdS)N)rangerreprr-print)r$cr+er%r%r&test2s r4__main__)r)rr'r-r4__name__r%r%r%r&s  __pycache__/conv.cpython-36.pyc000064400000015570152531664540012372 0ustar003 \%@s2dZddlZddlmZmZGdddejZdS)aConvert graminit.[ch] spit out by pgen to Python code. Pgen is the Python parser generator. It is useful to quickly create a parser from a grammar file in Python's grammar notation. But I don't want my parsers to be written in C (yet), so I'm translating the parsing tables to Python data structures and writing a Python parse engine. Note that the token numbers are constants determined by the standard Python tokenizer. The standard token module defines these numbers and their names (the names are not used much). The token numbers are hardcoded into the Python tokenizer and into pgen. A Python implementation of the Python tokenizer is also available, in the standard tokenize module. On the other hand, symbol numbers (representing the grammar's non-terminals) are assigned by pgen based on the actual grammar input. Note: this module is pretty much obsolete; the pgen module generates equivalent grammar tables directly from the Grammar.txt input file without having to invoke the Python pgen C program. N)grammartokenc@s0eZdZdZddZddZddZdd Zd S) Convertera2Grammar subclass that reads classic pgen output files. The run() method reads the tables as produced by the pgen parser generator, typically contained in two C files, graminit.h and graminit.c. The other methods are for internal use only. See the base class for more documentation. cCs |j||j||jdS)z@rPd||d|<qPWq6W||f||<qW|dt|}}|d kst||f||_g}|dt|}}tjd|}|st||ft |jd}x|t |D]p}|dt|}}tjd|}|s>t||f|j \}}t |}|dkrbd}nt|}|j ||fq W|dt|}}|d kst||f||_|dt|}}|dkst||f|dt|}}tjd|}|st||ft |jd}|t|jks&t|dt|}}|dksNt||f|dt|}}tjd|}|s~t||ft |jd}|t|jkst||f|dt|}}tjd|}|st||ft |jd} | |jkst||f| |_|dt|}}|d ks,t||fy|dt|}}Wntk rXYnXdslt||fdS)aParse the .c file written by pgen. (Internal) The file looks as follows. The first two lines are always this: #include "pgenheaders.h" #include "grammar.h" After that come four blocks: 1) one or more state definitions 2) a table defining dfas 3) a table defining labels 4) a struct defining the grammar A state definition has the following form: - one or more arc arrays, each of the form: static arc arcs__[] = { {, }, ... }; - followed by a state array, of the form: static state states_[] = { {, arcs__}, ... }; zCan't open %s: %sFNrr z#include "pgenheaders.h" z#include "grammar.h" z static arc z)static arc arcs_(\d+)_(\d+)\[(\d+)\] = {$z\s+{(\d+), (\d+)},$z}; z'static state states_(\d+)\[(\d+)\] = {$z\s+{(\d+), arcs_(\d+)_(\d+)},$zstatic dfa dfas\[(\d+)\] = {$z0\s+{(\d+), "(\w+)", (\d+), (\d+), states_(\d+),$z\s+("(?:\\\d\d\d)*")},$z!static label labels\[(\d+)\] = {$z\s+{(\d+), (0|"\w+")},$0zgrammar _PyParser_Grammar = { z \s+(\d+),$z dfas, z\s+{(\d+), labels},$z \s+(\d+)$)r rrnextr startswithrrlistmaprrrangeappendlenstatesgrouprreval enumerateorddfaslabelsstart StopIteration)!rrrrrrZallarcsr-rnmkZarcs_ijststater2ZndfasrrxyzfirstZ rawbitsetcZbyter3Znlabelsr4r r r rTs         "        zConverter.parse_graminit_ccCs\i|_i|_xJt|jD]<\}\}}|tjkrB|dk rB||j|<q|dkr||j|<qWdS)z1Create additional useful structures. (Internal).N)keywordstokensr0r3rNAME)rZilabeltypevaluer r r rs zConverter.finish_offN)__name__ __module__ __qualname____doc__r rrrr r r r r$s  &r)rLrZpgen2rrZGrammarrr r r r s__pycache__/grammar.cpython-36.pyc000064400000015621152531664540013050 0ustar003 \@sxdZddlZddlZddlmZmZGdddeZddZd Z iZ x.e j D]"Z e rNe j \ZZeeee e<qNWdS) aThis module defines the data structures used to represent a grammar. These are a bit arcane because they are derived from the data structures used by Python's 'pgen' parser generator. There's also a table here mapping operators to their names in the token module; the Python tokenize module reports all operators as the fallback token code OP, but the parser needs the actual token code. N)tokentokenizec@s@eZdZdZddZddZddZdd Zd d Zd d Z dS)Grammara Pgen parsing tables conversion class. Once initialized, this class supplies the grammar tables for the parsing engine implemented by parse.py. The parsing engine accesses the instance variables directly. The class here does not provide initialization of the tables; several subclasses exist to do this (see the conv and pgen modules). The load() method reads the tables from a pickle file, which is much faster than the other ways offered by subclasses. The pickle file is written by calling dump() (after loading the grammar tables using a subclass). The report() method prints a readable representation of the tables to stdout, for debugging. The instance variables are as follows: symbol2number -- a dict mapping symbol names to numbers. Symbol numbers are always 256 or higher, to distinguish them from token numbers, which are between 0 and 255 (inclusive). number2symbol -- a dict mapping numbers to symbol names; these two are each other's inverse. states -- a list of DFAs, where each DFA is a list of states, each state is a list of arcs, and each arc is a (i, j) pair where i is a label and j is a state number. The DFA number is the index into this list. (This name is slightly confusing.) Final states are represented by a special arc of the form (0, j) where j is its own state number. dfas -- a dict mapping symbol numbers to (DFA, first) pairs, where DFA is an item from the states list above, and first is a set of tokens that can begin this grammar rule (represented by a dict whose values are always 1). labels -- a list of (x, y) pairs where x is either a token number or a symbol number, and y is either None or a string; the strings are keywords. The label number is the index in this list; label numbers are used to mark state transitions (arcs) in the DFAs. start -- the number of the grammar's start symbol. keywords -- a dict mapping keyword strings to arc labels. tokens -- a dict mapping token numbers to arc labels. cCs<i|_i|_g|_i|_dg|_i|_i|_i|_d|_dS)NrEMPTY)rr) symbol2number number2symbolstatesdfaslabelskeywordstokens symbol2labelstart)selfr-/usr/lib64/python3.6/lib2to3/pgen2/grammar.py__init__MszGrammar.__init__c Cs2t|d}t|j}tj||dWdQRXdS)aDump the grammar tables to a pickle file. dump() recursively changes all dict to OrderedDict, so the pickled file is not exactly the same as what was passed in to dump(). load() uses the pickled file to create the tables, but only changes OrderedDict to dict at the top level; it does not recursively change OrderedDict to dict. So, the loaded tables are different from the original tables that were passed to load() in that some of the OrderedDict (from the pickled file) are not changed back to dict. For parsing, this has no effect on performance because OrderedDict uses dict's __getitem__ with nothing in between. wbN)open_make_deterministic__dict__pickledump)rfilenamefdrrrrXs  z Grammar.dumpc Cs0t|d}tj|}WdQRX|jj|dS)z+Load the grammar tables from a pickle file.rbN)rrloadrupdate)rrrrrrrr is z Grammar.loadcCs|jjtj|dS)z3Load the grammar tables from a pickle bytes object.N)rr!rloads)rZpklrrrr"osz Grammar.loadscCsX|j}x"dD]}t||t||jqW|jdd|_|jdd|_|j|_|S) z# Copy the grammar. rr r r rrN)rr r r rr) __class__setattrgetattrcopyr r r)rnewZ dict_attrrrrr&ssz Grammar.copycCsvddlm}td||jtd||jtd||jtd||jtd||jtd|jd S) z:Dump the grammar tables to standard output, for debugging.r)pprintZs2nZn2sr r r rN)r(printrr r r r r)rr(rrrreports      zGrammar.reportN) __name__ __module__ __qualname____doc__rrr r"r&r*rrrrrs4  rcCs^t|tr&tjtdd|jDSt|tr>dd|DSt|trZtdd|DS|S)Ncss|]\}}|t|fVqdS)N)r).0kvrrr sz&_make_deterministic..cSsg|] }t|qSr)r)r/errr sz'_make_deterministic..css|]}t|VqdS)N)r)r/r3rrrr2s) isinstancedict collections OrderedDictsorteditemslisttuple)toprrrrs   ra ( LPAR ) RPAR [ LSQB ] RSQB : COLON , COMMA ; SEMI + PLUS - MINUS * STAR / SLASH | VBAR & AMPER < LESS > GREATER = EQUAL . DOT % PERCENT ` BACKQUOTE { LBRACE } RBRACE @ AT @= ATEQUAL == EQEQUAL != NOTEQUAL <> NOTEQUAL <= LESSEQUAL >= GREATEREQUAL ~ TILDE ^ CIRCUMFLEX << LEFTSHIFT >> RIGHTSHIFT ** DOUBLESTAR += PLUSEQUAL -= MINEQUAL *= STAREQUAL /= SLASHEQUAL %= PERCENTEQUAL &= AMPEREQUAL |= VBAREQUAL ^= CIRCUMFLEXEQUAL <<= LEFTSHIFTEQUAL >>= RIGHTSHIFTEQUAL **= DOUBLESTAREQUAL // DOUBLESLASH //= DOUBLESLASHEQUAL -> RARROW )r.r7rrrobjectrrZ opmap_rawZopmap splitlineslinesplitopnamer%rrrr sy= __pycache__/__init__.cpython-36.pyc000064400000000237152531664540013156 0ustar003 \@sdZdS)zThe pgen2 package.N)__doc__rr./usr/lib64/python3.6/lib2to3/pgen2/__init__.pys__pycache__/driver.cpython-36.pyc000064400000012010152531664540012702 0ustar003 \@sdZdZddgZddlZddlZddlZddlZddlZddlZddl m Z m Z m Z m Z mZGdddeZd d ZdddZddZddZddZedkrejee dS)zZParser driver. This provides a high-level interface to parse a file into a syntax tree. z#Guido van Rossum Driver load_grammarN)grammarparsetokentokenizepgenc@sHeZdZdddZdddZdddZdd d Zdd d Zdd dZdS)rNcCs&||_|dkrtj}||_||_dS)N)rlogging getLoggerloggerconvert)selfrr r r,/usr/lib64/python3.6/lib2to3/pgen2/driver.py__init__ s zDriver.__init__FcCstj|j|j}|jd}d}d}}}} } d} xR|D]4} | \}}}} } |||fkr||f|ks|t||f|f|\} }|| kr| d| |7} | }d}||kr| | ||7} |}|tjtjfkr| |7} | \}}|j dr@|d7}d}q@|t j krtj |}|r,|j jdt j||| |j||| |frT|rR|j jdPd} | \}}|j dr@|d7}d}q@Wtjd||| |f|jS) z4Parse a series of tokens and return the syntax tree.rrN z%s %r (prefix=%r)zStop.zincomplete input)rZParserrr ZsetupAssertionErrorrCOMMENTNLendswithrOPZopmapr debugtok_nameZaddtokenZ ParseErrorZrootnode)rtokensrplinenocolumntypevaluestartendZ line_textprefixZ quintupleZs_linenoZs_columnrrr parse_tokens'sT      zDriver.parse_tokenscCstj|j}|j||S)z*Parse a stream and return the syntax tree.)rgenerate_tokensreadliner$)rstreamrrrrrparse_stream_rawWs zDriver.parse_stream_rawcCs |j||S)z*Parse a stream and return the syntax tree.)r()rr'rrrr parse_stream\szDriver.parse_streamc Cs*tj|d|}z |j||S|jXdS)z(Parse a file and return the syntax tree.rN)codecsopenr)close)rfilenameencodingrr'rrr parse_file`s zDriver.parse_filecCstjtj|j}|j||S)z*Parse a string and return the syntax tree.)rr%ioStringIOr&r$)rtextrrrrr parse_stringhszDriver.parse_string)NN)F)F)F)NF)F) __name__ __module__ __qualname__rr$r(r)r0r4rrrrrs   0   cCs:tjj|\}}|dkrd}||djtttjdS)Nz.txtr.z.pickle)ospathsplitextjoinmapstrsys version_info)gtheadtailrrr_generate_pickle_namensrD Grammar.txtTFcCs|dkrtj}|dkr t|n|}|s4t|| r|jd|tj|}|r|jd|y|j|Wqtk r}z|jd|WYdd}~XqXnt j }|j ||S)z'Load the grammar (maybe from a pickle).Nz!Generating grammar tables from %szWriting grammar tables to %szWriting failed: %s) r r rD_newerinfor Zgenerate_grammardumpOSErrorrGrammarload)rAZgpsaveforcer gerrrrus     cCs8tjj|sdStjj|s dStjj|tjj|kS)z0Inquire whether file a was written since file b.FT)r9r:existsgetmtime)abrrrrFs   rFcCsFtjj|rt|Sttjj|}tj||}tj }|j ||S)aNormally, loads a pickled grammar by doing pkgutil.get_data(package, pickled_grammar) where *pickled_grammar* is computed from *grammar_source* by adding the Python version and using a ``.pickle`` extension. However, if *grammar_source* is an extant file, load_grammar(grammar_source) is called instead. This facilitates using a packaged grammar file when needed but preserves load_grammar's automatic regeneration behavior when possible. ) r9r:isfilerrDbasenamepkgutilget_datarrJloads)packageZgrammar_sourceZ pickled_namedatarNrrrload_packaged_grammars   r[cGsF|stjdd}tjtjtjddx|D]}t|dddq,WdS)zMain program, when run as a script: produce grammar pickle files. Calls load_grammar for each argument, a path to a grammar text file. rNz %(message)s)levelr'formatT)rLrM)r?argvr Z basicConfigINFOstdoutr)argsrArrrmains  rb__main__)rENTFN)__doc__ __author____all__r+r1r9r rVr?rrrrrr objectrrDrrFr[rbr5exitintrrrr s$P   __pycache__/tokenize.cpython-36.pyc000064400000036032152531664540013251 0ustar003 \NX=@sdZdZdZddlZddlZddlmZmZddlTddl m Z d d e e Dd d d gZ [ ye Wnek r~eZ YnXddZddZddZdZdZeedeeeZdZdZdZdZeddZeeeeeZdZeddeeZd eZeeeZed!ed"Z ee eeZ!d#Z"d$Z#d%Z$d&Z%d'Z&ee&d(e&d)Z'ee&d*e&d+Z(ed,d-d.d/d0d1d2d3d4 Z)d5Z*ed6d7Z+ee)e*e+Z,ee!e,e(eZ-ee-Z.ee&d8ed9de&d:ed;dZ/edee'Z0eee0e!e,e/eZ1e2e3ej4e.e1e$e%f\Z5Z6Z7Z8ej4e"ej4e#e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8e7e8ddddddddd<4Z9iZ:xdD]Z;e;e:e;<qWiZZ?Gddde>Z@ddZAeAfdd ZBddZCGdddZDej4dejEZFej4dejEZGddZHddZIdd ZJdd ZKeLdkrddlMZMeNeMjOdkr|eBePeMjOdjQn eBeMjRjQdS)aTokenization help for Python programs. generate_tokens(readline) is a generator that breaks a stream of text into Python tokens. It accepts a readline-like method which is called repeatedly to get the next line of input (or "" for EOF). It generates 5-tuples with these members: the token type (see token.py) the token (a string) the starting (row, column) indices of the token (a 2-tuple of ints) the ending (row, column) indices of the token (a 2-tuple of ints) the original line (string) It is designed to match the working of the Python tokenizer exactly, except that it produces COMMENT tokens for comments and gives type OP for all operators Older entry points tokenize_loop(readline, tokeneater) tokenize(readline, tokeneater=printtoken) are the same, except instead of generating tokens, tokeneater is a callback function to which the 5 fields described above are passed as 5 arguments, each time a new token is found.zKa-Ping Yee z@GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip MontanaroN)BOM_UTF8lookup)*)tokencCsg|]}|ddkr|qS)r_).0xrr./usr/lib64/python3.6/lib2to3/pgen2/tokenize.py %sr tokenizegenerate_tokens untokenizecGsddj|dS)N(|))join)choicesrrr group0srcGs t|dS)Nr)r)rrrr any1srcGs t|dS)N?)r)rrrr maybe2srz[ \f\t]*z #[^\r\n]*z\\\r?\nz [a-zA-Z_]\w*z0[bB]_?[01]+(?:_[01]+)*z(0[xX]_?[\da-fA-F]+(?:_[\da-fA-F]+)*[lL]?z0[oO]?_?[0-7]+(?:_[0-7]+)*[lL]?z[1-9]\d*(?:_\d+)*[lL]?z0[lL]?z[eE][-+]?\d+(?:_\d+)*z\d+(?:_\d+)*\.(?:\d+(?:_\d+)*)?z\.\d+(?:_\d+)*z \d+(?:_\d+)*z\d+(?:_\d+)*[jJ]z[jJ]z[^'\\]*(?:\\.[^'\\]*)*'z[^"\\]*(?:\\.[^"\\]*)*"z%[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''z%[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""z#(?:[uUrRbBfF]|[rR][bB]|[bBuU][rR])?'''"""z'[^\n'\\]*(?:\\.[^\n'\\]*)*'z"[^\n"\\]*(?:\\.[^\n"\\]*)*"z\*\*=?z>>=?z<<=?z<>z!=z//=?z->z[+\-*/%&@|^=<>]=?~z[][(){}]z\r?\nz[:;.,`@]z'[^\n'\\]*(?:\\.[^\n'\\]*)*'z"[^\n"\\]*(?:\\.[^\n"\\]*)*")4rrz'''z"""zr'''zr"""zu'''zu"""zb'''zb"""zf'''zf"""zur'''zur"""zbr'''zbr"""zrb'''zrb"""zR'''zR"""zU'''zU"""zB'''zB"""zF'''zF"""zuR'''zuR"""zUr'''zUr"""zUR'''zUR"""zbR'''zbR"""zBr'''zBr"""zBR'''zBR"""zrB'''zrB"""zRb'''zRb"""zRB'''zRB"""rRuUfFbBr'''r"""R'''R"""u'''u"""U'''U"""b'''b"""B'''B"""f'''f"""F'''F"""ur'''ur"""Ur'''Ur"""uR'''uR"""UR'''UR"""br'''br"""Br'''Br"""bR'''bR"""BR'''BR"""rb'''rb"""Rb'''Rb"""rB'''rB"""RB'''RB"""r'r"R'R"u'u"U'U"b'b"B'B"f'f"F'F"ur'ur"Ur'Ur"uR'uR"UR'UR"br'br"Br'Br"bR'bR"BR'BR"rb'rb"Rb'Rb"rB'rB"RB'RB"c@s eZdZdS) TokenErrorN)__name__ __module__ __qualname__rrrr rwsrwc@s eZdZdS)StopTokenizingN)rxryrzrrrr r{sr{c Cs4|\}}|\}}td||||t|t|fdS)Nz%d,%d-%d,%d: %s %s)printtok_namerepr) typerZxxx_todo_changemeZxxx_todo_changeme1lineZsrowZscolZerowZecolrrr printtokensrc Cs(yt||Wntk r"YnXdS)a: The tokenize() function accepts two parameters: one representing the input stream, and one providing an output mechanism for tokenize(). The first parameter, readline, must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string. The second parameter, tokeneater, must also be a callable object. It is called once for each token, with five arguments, corresponding to the tuples generated by generate_tokens(). N) tokenize_loopr{)readline tokeneaterrrr r s cCsxt|D] }||q WdS)N)r)rrZ token_inforrr rsrc@s,eZdZddZddZddZddZd S) UntokenizercCsg|_d|_d|_dS)Nrr)tokensprev_rowprev_col)selfrrr __init__szUntokenizer.__init__cCs8|\}}||jkst||j}|r4|jjd|dS)N )rAssertionErrorrrappend)rstartrowcol col_offsetrrr add_whitespaces  zUntokenizer.add_whitespacecCsxv|D]n}t|dkr$|j||P|\}}}}}|j||jj||\|_|_|ttfkr|jd7_d|_qWdj |jS)Nrr) lencompatrrrrrNEWLINENLr)riterablettok_typerrendrrrr rs        zUntokenizer.untokenizec Csd}g}|jj}|\}}|ttfkr,|d7}|ttfkr|t kr|j qBn*|ttfkrd}n|r|r||dd}||qBWdS)NFrTrr) rrNAMENUMBERrrASYNCAWAITINDENTDEDENTpop) rrr startlineindents toks_appendtoknumtokvaltokrrr rs0      zUntokenizer.compatN)rxryrzrrrrrrrr rsrz&^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)s^[ \t\f]*(?:[#\r\n]|$)cCsH|ddjjdd}|dks*|jdr.dS|d ks@|jdrDdS|S)z(Imitates get_normal_name in tokenizer.c.N r-zutf-8zutf-8-latin-1 iso-8859-1 iso-latin-1latin-1- iso-8859-1- iso-latin-1-)rrr)rrr)lowerreplace startswith)orig_encencrrr _get_normal_names rcsdd}d}fdd}fdd}|}|jtrHd|d d}d }|sT|gfS||}|rj||gfStj|s~||gfS|}|s||gfS||}|r|||gfS|||gfS) a The detect_encoding() function is used to detect the encoding that should be used to decode a Python source file. It requires one argument, readline, in the same way as the tokenize() generator. It will call readline a maximum of twice, and return the encoding used (as a string) and a list of any lines (left as bytes) it has read in. It detects the encoding from the presence of a utf-8 bom or an encoding cookie as specified in pep-0263. If both a bom and a cookie are present, but disagree, a SyntaxError will be raised. If the encoding cookie is an invalid charset, raise a SyntaxError. Note that if a utf-8 bom is found, 'utf-8-sig' is returned. If no encoding is specified, then the default of 'utf-8' will be returned. FNzutf-8c s"yStk rtSXdS)N) StopIterationbytesr)rrr read_or_stop sz%detect_encoding..read_or_stopcsy|jd}Wntk r"dSXtj|}|s6dSt|jd}y t|}Wn tk rptd|YnXr|j dkrtd|d7}|S)Nasciirzunknown encoding: zutf-8zencoding problem: utf-8z-sig) decodeUnicodeDecodeError cookie_rematchrrr LookupError SyntaxErrorname)r line_stringrencodingcodec) bom_foundrr find_cookie&s"   z$detect_encoding..find_cookieTz utf-8-sig)rrblank_rer)rrdefaultrrfirstsecondr)rrr detect_encoding s0         rcCst}|j|S)aTransform tokens back into Python source code. Each element returned by the iterable must be a token sequence with at least two elements, a token number and token value. If only two tokens are passed, the resulting output is poor. Round-trip invariant for full input: Untokenized source will match input source exactly Round-trip invariant for limited intput: # Output text will tokenize the back to the input t1 = [tok[:2] for tok in generate_tokens(f.readline)] newcode = untokenize(t1) readline = iter(newcode.splitlines(1)).next t2 = [tok[:2] for tokin generate_tokens(readline)] assert t1 == t2 )rr)rutrrr rTsc!csd}}}tjdd}}d \}}d}dg} d} d} d} d} xy |}Wntk rdd}YnX|d}dt|}}|rF|std||j|}|r|jd}}t||d||||f||fVd!\}}d}nd|r0|d"dd kr0|d#dd kr0t||||t|f|fVd}d}qBn||}||}qBnF|dkrt| rt|s`Pd}xf||kr||d kr|d}n6||dkr|t dt }n||dkrd}nP|d}qfW||krP| r| Vd} ||dkr||dkrh||dj d}|t|}t |||f||t|f|fVt ||d||f|t|f|fVqBt t f||dk||d||f|t|f|fVqB|| d$kr| j |t|d||df||f|fVxt|| d%krJ|| krtdd|||f| dd&} | r.| | d'kr.d} d} d} td||f||f|fVqW| r| r| | d(krd} d} d} n|std|dfd}x||krJtj||}|r|jd\}}||f||f|}}}|||||}}||ks|dkr|dkrt||||fVqF|dkrft}|dkr8t }n | rBd} | rR| Vd} |||||fVqF|dkr|jd st| r| Vd} t ||||fVqF|tkr$t|}|j||}|r|jd}|||}| r| Vd} t||||f|fVn||f}||d}|}PqF|tksR|dd tksR|dd tkr|d)dkr||f}t|pt|dpt|d }||dd}}|}Pn | r| Vd} t||||fVqF||kr|d*kr| r|dkrtnt||||fVqt||||f}|dkr.| r.|} q|dkr| r| dtkr| ddkrd} | d+} t| d| d | d | dfVd} | r| Vd} |Vnz|dkr| r| Vd} t ||||f|fVd}nF|dkr|d}n|dkr|d}| r | Vd} t||||fVn(t||||f||df|fV|d}qWqBW| r`| Vd} x.| ddD]} td|df|dfdfVqnWtd|df|dfdfVdS),aT The generate_tokens() generator requires one argument, readline, which must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string. Alternately, readline can be a callable function terminating with StopIteration: readline = open(myfile).next # Example of alternate readline The generator produces 5-tuples with these members: the token type; the token string; a 2-tuple (srow, scol) of ints specifying the row and column where the token begins in the source; a 2-tuple (erow, ecol) of ints specifying the row and column where the token ends in the source; and the line on which the token was found. The line passed is the logical line; continuation lines are included. rr 0123456789rNFrzEOF in multi-line stringrz\ rz\ r  z# #z z3unindent does not match any outer indentation levelz zEOF in multi-line statement.T asyncawaitdef\z([{z)]})rr)rrrrrrrr)rrr)stringZ ascii_lettersrrrwrrSTRING ERRORTOKENtabsizerstripCOMMENTrrrIndentationErrorr pseudoprogspanrrendswithr triple_quotedendprogs single_quotedrrrOP ENDMARKER)!rlnumparenlev continuedZ namecharsnumcharscontstrneedcontcontlinerstashed async_defasync_def_indent async_def_nlrposmaxstrstartendprogendmatchrcolumn comment_tokennl_pos pseudomatchrsposeposrinitialnewlinerindentrrr risr     *                             __main__)*rrr&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrM)*rrrNrOrPrQrRrSrTrUrVrWrXrYrZr[r\r]r^r_r`rarbrcrdrerfrgrhrirjrkrlrmrnrorprqrrrsrtru)S__doc__ __author__ __credits__rrecodecsrrZlib2to3.pgen2.tokenrrdir__all__r NameErrorstrrrr WhitespaceCommentIgnoreName Binnumber Hexnumber Octnumber Decnumber IntnumberExponent PointfloatExpfloat Floatnumber ImagnumberNumberSingleDoubleSingle3Double3Z _litprefixTripleStringOperatorBracketSpecialFunny PlainTokenTokenContStr PseudoExtras PseudoTokenlistmapcompileZ tokenprogrZ single3progZ double3progrrrrr Exceptionrwr{rr rrASCIIrrrrrrrxsysrargvopenrstdinrrrr s              8 Ic