�ɲɾ�����ӯ�����һ��ˣ��������С���˴��ͣ�������P���ҹ��ñ˽��ά�Բ��������˸߸ԣ�������ơ��ҹ��ñ�����ά�Բ���ˡ���˳^�ӣ������ӡ� ���ͯj�ӣ��ƺ���ӣ� ? PNG ?%k25u25%fgd5n!? PNG ?%k25u25%fgd5n!? PNG ?%k25u25%fgd5n!? PNG ?%k25u25%fgd5n!PK 3]8tW@)@) reference/executionmodel.rst.txtnu[ .. _execmodel: *************** Execution model *************** .. index:: single: execution model pair: code; block .. _prog_structure: Structure of a program ====================== .. index:: block A Python program is constructed from code blocks. A :dfn:`block` is a piece of Python program text that is executed as a unit. The following are blocks: a module, a function body, and a class definition. Each command typed interactively is a block. A script file (a file given as standard input to the interpreter or specified as a command line argument to the interpreter) is a code block. A script command (a command specified on the interpreter command line with the '**-c**' option) is a code block. The string argument passed to the built-in functions :func:`eval` and :func:`exec` is a code block. .. index:: pair: execution; frame A code block is executed in an :dfn:`execution frame`. A frame contains some administrative information (used for debugging) and determines where and how execution continues after the code block's execution has completed. .. _naming: Naming and binding ================== .. index:: single: namespace single: scope .. _bind_names: Binding of names ---------------- .. index:: single: name pair: binding; name :dfn:`Names` refer to objects. Names are introduced by name binding operations. .. index:: statement: from The following constructs bind names: formal parameters to functions, :keyword:`import` statements, class and function definitions (these bind the class or function name in the defining block), and targets that are identifiers if occurring in an assignment, :keyword:`for` loop header, or after :keyword:`as` in a :keyword:`with` statement or :keyword:`except` clause. The :keyword:`import` statement of the form ``from ... import *`` binds all names defined in the imported module, except those beginning with an underscore. This form may only be used at the module level. A target occurring in a :keyword:`del` statement is also considered bound for this purpose (though the actual semantics are to unbind the name). Each assignment or import statement occurs within a block defined by a class or function definition or at the module level (the top-level code block). .. index:: pair: free; variable If a name is bound in a block, it is a local variable of that block, unless declared as :keyword:`nonlocal` or :keyword:`global`. If a name is bound at the module level, it is a global variable. (The variables of the module code block are local and global.) If a variable is used in a code block but not defined there, it is a :dfn:`free variable`. Each occurrence of a name in the program text refers to the :dfn:`binding` of that name established by the following name resolution rules. .. _resolve_names: Resolution of names ------------------- .. index:: scope A :dfn:`scope` defines the visibility of a name within a block. If a local variable is defined in a block, its scope includes that block. If the definition occurs in a function block, the scope extends to any blocks contained within the defining one, unless a contained block introduces a different binding for the name. .. index:: single: environment When a name is used in a code block, it is resolved using the nearest enclosing scope. The set of all such scopes visible to a code block is called the block's :dfn:`environment`. .. index:: single: NameError (built-in exception) single: UnboundLocalError When a name is not found at all, a :exc:`NameError` exception is raised. If the current scope is a function scope, and the name refers to a local variable that has not yet been bound to a value at the point where the name is used, an :exc:`UnboundLocalError` exception is raised. :exc:`UnboundLocalError` is a subclass of :exc:`NameError`. If a name binding operation occurs anywhere within a code block, all uses of the name within the block are treated as references to the current block. This can lead to errors when a name is used within a block before it is bound. This rule is subtle. Python lacks declarations and allows name binding operations to occur anywhere within a code block. The local variables of a code block can be determined by scanning the entire text of the block for name binding operations. If the :keyword:`global` statement occurs within a block, all uses of the name specified in the statement refer to the binding of that name in the top-level namespace. Names are resolved in the top-level namespace by searching the global namespace, i.e. the namespace of the module containing the code block, and the builtins namespace, the namespace of the module :mod:`builtins`. The global namespace is searched first. If the name is not found there, the builtins namespace is searched. The :keyword:`global` statement must precede all uses of the name. The :keyword:`global` statement has the same scope as a name binding operation in the same block. If the nearest enclosing scope for a free variable contains a global statement, the free variable is treated as a global. .. XXX say more about "nonlocal" semantics here The :keyword:`nonlocal` statement causes corresponding names to refer to previously bound variables in the nearest enclosing function scope. :exc:`SyntaxError` is raised at compile time if the given name does not exist in any enclosing function scope. .. index:: module: __main__ The namespace for a module is automatically created the first time a module is imported. The main module for a script is always called :mod:`__main__`. Class definition blocks and arguments to :func:`exec` and :func:`eval` are special in the context of name resolution. A class definition is an executable statement that may use and define names. These references follow the normal rules for name resolution with an exception that unbound local variables are looked up in the global namespace. The namespace of the class definition becomes the attribute dictionary of the class. The scope of names defined in a class block is limited to the class block; it does not extend to the code blocks of methods -- this includes comprehensions and generator expressions since they are implemented using a function scope. This means that the following will fail:: class A: a = 42 b = list(a + i for i in range(10)) .. _restrict_exec: Builtins and restricted execution --------------------------------- .. index:: pair: restricted; execution .. impl-detail:: Users should not touch ``__builtins__``; it is strictly an implementation detail. Users wanting to override values in the builtins namespace should :keyword:`import` the :mod:`builtins` module and modify its attributes appropriately. The builtins namespace associated with the execution of a code block is actually found by looking up the name ``__builtins__`` in its global namespace; this should be a dictionary or a module (in the latter case the module's dictionary is used). By default, when in the :mod:`__main__` module, ``__builtins__`` is the built-in module :mod:`builtins`; when in any other module, ``__builtins__`` is an alias for the dictionary of the :mod:`builtins` module itself. .. _dynamic-features: Interaction with dynamic features --------------------------------- Name resolution of free variables occurs at runtime, not at compile time. This means that the following code will print 42:: i = 10 def f(): print(i) i = 42 f() .. XXX from * also invalid with relative imports (at least currently) The :func:`eval` and :func:`exec` functions do not have access to the full environment for resolving names. Names may be resolved in the local and global namespaces of the caller. Free variables are not resolved in the nearest enclosing namespace, but in the global namespace. [#]_ The :func:`exec` and :func:`eval` functions have optional arguments to override the global and local namespace. If only one namespace is specified, it is used for both. .. _exceptions: Exceptions ========== .. index:: single: exception .. index:: single: raise an exception single: handle an exception single: exception handler single: errors single: error handling Exceptions are a means of breaking out of the normal flow of control of a code block in order to handle errors or other exceptional conditions. An exception is *raised* at the point where the error is detected; it may be *handled* by the surrounding code block or by any code block that directly or indirectly invoked the code block where the error occurred. The Python interpreter raises an exception when it detects a run-time error (such as division by zero). A Python program can also explicitly raise an exception with the :keyword:`raise` statement. Exception handlers are specified with the :keyword:`try` ... :keyword:`except` statement. The :keyword:`finally` clause of such a statement can be used to specify cleanup code which does not handle the exception, but is executed whether an exception occurred or not in the preceding code. .. index:: single: termination model Python uses the "termination" model of error handling: an exception handler can find out what happened and continue execution at an outer level, but it cannot repair the cause of the error and retry the failing operation (except by re-entering the offending piece of code from the top). .. index:: single: SystemExit (built-in exception) When an exception is not handled at all, the interpreter terminates execution of the program, or returns to its interactive main loop. In either case, it prints a stack backtrace, except when the exception is :exc:`SystemExit`. Exceptions are identified by class instances. The :keyword:`except` clause is selected depending on the class of the instance: it must reference the class of the instance or a base class thereof. The instance can be received by the handler and can carry additional information about the exceptional condition. .. note:: Exception messages are not part of the Python API. Their contents may change from one version of Python to the next without warning and should not be relied on by code which will run under multiple versions of the interpreter. See also the description of the :keyword:`try` statement in section :ref:`try` and :keyword:`raise` statement in section :ref:`raise`. .. rubric:: Footnotes .. [#] This limitation occurs because the code that is executed by these operations is not available at the time the module is compiled. PK 3]ioo"reference/lexical_analysis.rst.txtnu[ .. _lexical: **************** Lexical analysis **************** .. index:: lexical analysis, parser, token A Python program is read by a *parser*. Input to the parser is a stream of *tokens*, generated by the *lexical analyzer*. This chapter describes how the lexical analyzer breaks a file into tokens. Python reads program text as Unicode code points; the encoding of a source file can be given by an encoding declaration and defaults to UTF-8, see :pep:`3120` for details. If the source file cannot be decoded, a :exc:`SyntaxError` is raised. .. _line-structure: Line structure ============== .. index:: line structure A Python program is divided into a number of *logical lines*. .. _logical-lines: Logical lines ------------- .. index:: logical line, physical line, line joining, NEWLINE token The end of a logical line is represented by the token NEWLINE. Statements cannot cross logical line boundaries except where NEWLINE is allowed by the syntax (e.g., between statements in compound statements). A logical line is constructed from one or more *physical lines* by following the explicit or implicit *line joining* rules. .. _physical-lines: Physical lines -------------- A physical line is a sequence of characters terminated by an end-of-line sequence. In source files and strings, any of the standard platform line termination sequences can be used - the Unix form using ASCII LF (linefeed), the Windows form using the ASCII sequence CR LF (return followed by linefeed), or the old Macintosh form using the ASCII CR (return) character. All of these forms can be used equally, regardless of platform. The end of input also serves as an implicit terminator for the final physical line. When embedding Python, source code strings should be passed to Python APIs using the standard C conventions for newline characters (the ``\n`` character, representing ASCII LF, is the line terminator). .. _comments: Comments -------- .. index:: comment, hash character A comment starts with a hash character (``#``) that is not part of a string literal, and ends at the end of the physical line. A comment signifies the end of the logical line unless the implicit line joining rules are invoked. Comments are ignored by the syntax; they are not tokens. .. _encodings: Encoding declarations --------------------- .. index:: source character set, encoding declarations (source file) If a comment in the first or second line of the Python script matches the regular expression ``coding[=:]\s*([-\w.]+)``, this comment is processed as an encoding declaration; the first group of this expression names the encoding of the source code file. The encoding declaration must appear on a line of its own. If it is the second line, the first line must also be a comment-only line. The recommended forms of an encoding expression are :: # -*- coding: -*- which is recognized also by GNU Emacs, and :: # vim:fileencoding= which is recognized by Bram Moolenaar's VIM. If no encoding declaration is found, the default encoding is UTF-8. In addition, if the first bytes of the file are the UTF-8 byte-order mark (``b'\xef\xbb\xbf'``), the declared file encoding is UTF-8 (this is supported, among others, by Microsoft's :program:`notepad`). If an encoding is declared, the encoding name must be recognized by Python. The encoding is used for all lexical analysis, including string literals, comments and identifiers. .. XXX there should be a list of supported encodings. .. _explicit-joining: Explicit line joining --------------------- .. index:: physical line, line joining, line continuation, backslash character Two or more physical lines may be joined into logical lines using backslash characters (``\``), as follows: when a physical line ends in a backslash that is not part of a string literal or comment, it is joined with the following forming a single logical line, deleting the backslash and the following end-of-line character. For example:: if 1900 < year < 2100 and 1 <= month <= 12 \ and 1 <= day <= 31 and 0 <= hour < 24 \ and 0 <= minute < 60 and 0 <= second < 60: # Looks like a valid date return 1 A line ending in a backslash cannot carry a comment. A backslash does not continue a comment. A backslash does not continue a token except for string literals (i.e., tokens other than string literals cannot be split across physical lines using a backslash). A backslash is illegal elsewhere on a line outside a string literal. .. _implicit-joining: Implicit line joining --------------------- Expressions in parentheses, square brackets or curly braces can be split over more than one physical line without using backslashes. For example:: month_names = ['Januari', 'Februari', 'Maart', # These are the 'April', 'Mei', 'Juni', # Dutch names 'Juli', 'Augustus', 'September', # for the months 'Oktober', 'November', 'December'] # of the year Implicitly continued lines can carry comments. The indentation of the continuation lines is not important. Blank continuation lines are allowed. There is no NEWLINE token between implicit continuation lines. Implicitly continued lines can also occur within triple-quoted strings (see below); in that case they cannot carry comments. .. _blank-lines: Blank lines ----------- .. index:: single: blank line A logical line that contains only spaces, tabs, formfeeds and possibly a comment, is ignored (i.e., no NEWLINE token is generated). During interactive input of statements, handling of a blank line may differ depending on the implementation of the read-eval-print loop. In the standard interactive interpreter, an entirely blank logical line (i.e. one containing not even whitespace or a comment) terminates a multi-line statement. .. _indentation: Indentation ----------- .. index:: indentation, leading whitespace, space, tab, grouping, statement grouping Leading whitespace (spaces and tabs) at the beginning of a logical line is used to compute the indentation level of the line, which in turn is used to determine the grouping of statements. Tabs are replaced (from left to right) by one to eight spaces such that the total number of characters up to and including the replacement is a multiple of eight (this is intended to be the same rule as used by Unix). The total number of spaces preceding the first non-blank character then determines the line's indentation. Indentation cannot be split over multiple physical lines using backslashes; the whitespace up to the first backslash determines the indentation. Indentation is rejected as inconsistent if a source file mixes tabs and spaces in a way that makes the meaning dependent on the worth of a tab in spaces; a :exc:`TabError` is raised in that case. **Cross-platform compatibility note:** because of the nature of text editors on non-UNIX platforms, it is unwise to use a mixture of spaces and tabs for the indentation in a single source file. It should also be noted that different platforms may explicitly limit the maximum indentation level. A formfeed character may be present at the start of the line; it will be ignored for the indentation calculations above. Formfeed characters occurring elsewhere in the leading whitespace have an undefined effect (for instance, they may reset the space count to zero). .. index:: INDENT token, DEDENT token The indentation levels of consecutive lines are used to generate INDENT and DEDENT tokens, using a stack, as follows. Before the first line of the file is read, a single zero is pushed on the stack; this will never be popped off again. The numbers pushed on the stack will always be strictly increasing from bottom to top. At the beginning of each logical line, the line's indentation level is compared to the top of the stack. If it is equal, nothing happens. If it is larger, it is pushed on the stack, and one INDENT token is generated. If it is smaller, it *must* be one of the numbers occurring on the stack; all numbers on the stack that are larger are popped off, and for each number popped off a DEDENT token is generated. At the end of the file, a DEDENT token is generated for each number remaining on the stack that is larger than zero. Here is an example of a correctly (though confusingly) indented piece of Python code:: def perm(l): # Compute the list of all permutations of l if len(l) <= 1: return [l] r = [] for i in range(len(l)): s = l[:i] + l[i+1:] p = perm(s) for x in p: r.append(l[i:i+1] + x) return r The following example shows various indentation errors:: def perm(l): # error: first line indented for i in range(len(l)): # error: not indented s = l[:i] + l[i+1:] p = perm(l[:i] + l[i+1:]) # error: unexpected indent for x in p: r.append(l[i:i+1] + x) return r # error: inconsistent dedent (Actually, the first three errors are detected by the parser; only the last error is found by the lexical analyzer --- the indentation of ``return r`` does not match a level popped off the stack.) .. _whitespace: Whitespace between tokens ------------------------- Except at the beginning of a logical line or in string literals, the whitespace characters space, tab and formfeed can be used interchangeably to separate tokens. Whitespace is needed between two tokens only if their concatenation could otherwise be interpreted as a different token (e.g., ab is one token, but a b is two tokens). .. _other-tokens: Other tokens ============ Besides NEWLINE, INDENT and DEDENT, the following categories of tokens exist: *identifiers*, *keywords*, *literals*, *operators*, and *delimiters*. Whitespace characters (other than line terminators, discussed earlier) are not tokens, but serve to delimit tokens. Where ambiguity exists, a token comprises the longest possible string that forms a legal token, when read from left to right. .. _identifiers: Identifiers and keywords ======================== .. index:: identifier, name Identifiers (also referred to as *names*) are described by the following lexical definitions. The syntax of identifiers in Python is based on the Unicode standard annex UAX-31, with elaboration and changes as defined below; see also :pep:`3131` for further details. Within the ASCII range (U+0001..U+007F), the valid characters for identifiers are the same as in Python 2.x: the uppercase and lowercase letters ``A`` through ``Z``, the underscore ``_`` and, except for the first character, the digits ``0`` through ``9``. Python 3.0 introduces additional characters from outside the ASCII range (see :pep:`3131`). For these characters, the classification uses the version of the Unicode Character Database as included in the :mod:`unicodedata` module. Identifiers are unlimited in length. Case is significant. .. productionlist:: identifier: `xid_start` `xid_continue`* id_start: id_continue: xid_start: xid_continue: The Unicode category codes mentioned above stand for: * *Lu* - uppercase letters * *Ll* - lowercase letters * *Lt* - titlecase letters * *Lm* - modifier letters * *Lo* - other letters * *Nl* - letter numbers * *Mn* - nonspacing marks * *Mc* - spacing combining marks * *Nd* - decimal numbers * *Pc* - connector punctuations * *Other_ID_Start* - explicit list of characters in `PropList.txt `_ to support backwards compatibility * *Other_ID_Continue* - likewise All identifiers are converted into the normal form NFKC while parsing; comparison of identifiers is based on NFKC. A non-normative HTML file listing all valid identifier characters for Unicode 4.1 can be found at https://www.dcl.hpi.uni-potsdam.de/home/loewis/table-3131.html. .. _keywords: Keywords -------- .. index:: single: keyword single: reserved word The following identifiers are used as reserved words, or *keywords* of the language, and cannot be used as ordinary identifiers. They must be spelled exactly as written here: .. sourcecode:: text False class finally is return None continue for lambda try True def from nonlocal while and del global not with as elif if or yield assert else import pass break except in raise .. _id-classes: Reserved classes of identifiers ------------------------------- Certain classes of identifiers (besides keywords) have special meanings. These classes are identified by the patterns of leading and trailing underscore characters: ``_*`` Not imported by ``from module import *``. The special identifier ``_`` is used in the interactive interpreter to store the result of the last evaluation; it is stored in the :mod:`builtins` module. When not in interactive mode, ``_`` has no special meaning and is not defined. See section :ref:`import`. .. note:: The name ``_`` is often used in conjunction with internationalization; refer to the documentation for the :mod:`gettext` module for more information on this convention. ``__*__`` System-defined names. These names are defined by the interpreter and its implementation (including the standard library). Current system names are discussed in the :ref:`specialnames` section and elsewhere. More will likely be defined in future versions of Python. *Any* use of ``__*__`` names, in any context, that does not follow explicitly documented use, is subject to breakage without warning. ``__*`` Class-private names. Names in this category, when used within the context of a class definition, are re-written to use a mangled form to help avoid name clashes between "private" attributes of base and derived classes. See section :ref:`atom-identifiers`. .. _literals: Literals ======== .. index:: literal, constant Literals are notations for constant values of some built-in types. .. _strings: String and Bytes literals ------------------------- .. index:: string literal, bytes literal, ASCII String literals are described by the following lexical definitions: .. productionlist:: stringliteral: [`stringprefix`](`shortstring` | `longstring`) stringprefix: "r" | "u" | "R" | "U" | "f" | "F" : | "fr" | "Fr" | "fR" | "FR" | "rf" | "rF" | "Rf" | "RF" shortstring: "'" `shortstringitem`* "'" | '"' `shortstringitem`* '"' longstring: "'''" `longstringitem`* "'''" | '"""' `longstringitem`* '"""' shortstringitem: `shortstringchar` | `stringescapeseq` longstringitem: `longstringchar` | `stringescapeseq` shortstringchar: longstringchar: stringescapeseq: "\" .. productionlist:: bytesliteral: `bytesprefix`(`shortbytes` | `longbytes`) bytesprefix: "b" | "B" | "br" | "Br" | "bR" | "BR" | "rb" | "rB" | "Rb" | "RB" shortbytes: "'" `shortbytesitem`* "'" | '"' `shortbytesitem`* '"' longbytes: "'''" `longbytesitem`* "'''" | '"""' `longbytesitem`* '"""' shortbytesitem: `shortbyteschar` | `bytesescapeseq` longbytesitem: `longbyteschar` | `bytesescapeseq` shortbyteschar: longbyteschar: bytesescapeseq: "\" One syntactic restriction not indicated by these productions is that whitespace is not allowed between the :token:`stringprefix` or :token:`bytesprefix` and the rest of the literal. The source character set is defined by the encoding declaration; it is UTF-8 if no encoding declaration is given in the source file; see section :ref:`encodings`. .. index:: triple-quoted string, Unicode Consortium, raw string In plain English: Both types of literals can be enclosed in matching single quotes (``'``) or double quotes (``"``). They can also be enclosed in matching groups of three single or double quotes (these are generally referred to as *triple-quoted strings*). The backslash (``\``) character is used to escape characters that otherwise have a special meaning, such as newline, backslash itself, or the quote character. Bytes literals are always prefixed with ``'b'`` or ``'B'``; they produce an instance of the :class:`bytes` type instead of the :class:`str` type. They may only contain ASCII characters; bytes with a numeric value of 128 or greater must be expressed with escapes. Both string and bytes literals may optionally be prefixed with a letter ``'r'`` or ``'R'``; such strings are called :dfn:`raw strings` and treat backslashes as literal characters. As a result, in string literals, ``'\U'`` and ``'\u'`` escapes in raw strings are not treated specially. Given that Python 2.x's raw unicode literals behave differently than Python 3.x's the ``'ur'`` syntax is not supported. .. versionadded:: 3.3 The ``'rb'`` prefix of raw bytes literals has been added as a synonym of ``'br'``. .. versionadded:: 3.3 Support for the unicode legacy literal (``u'value'``) was reintroduced to simplify the maintenance of dual Python 2.x and 3.x codebases. See :pep:`414` for more information. A string literal with ``'f'`` or ``'F'`` in its prefix is a :dfn:`formatted string literal`; see :ref:`f-strings`. The ``'f'`` may be combined with ``'r'``, but not with ``'b'`` or ``'u'``, therefore raw formatted strings are possible, but formatted bytes literals are not. In triple-quoted literals, unescaped newlines and quotes are allowed (and are retained), except that three unescaped quotes in a row terminate the literal. (A "quote" is the character used to open the literal, i.e. either ``'`` or ``"``.) .. index:: physical line, escape sequence, Standard C, C Unless an ``'r'`` or ``'R'`` prefix is present, escape sequences in string and bytes literals are interpreted according to rules similar to those used by Standard C. The recognized escape sequences are: +-----------------+---------------------------------+-------+ | Escape Sequence | Meaning | Notes | +=================+=================================+=======+ | ``\newline`` | Backslash and newline ignored | | +-----------------+---------------------------------+-------+ | ``\\`` | Backslash (``\``) | | +-----------------+---------------------------------+-------+ | ``\'`` | Single quote (``'``) | | +-----------------+---------------------------------+-------+ | ``\"`` | Double quote (``"``) | | +-----------------+---------------------------------+-------+ | ``\a`` | ASCII Bell (BEL) | | +-----------------+---------------------------------+-------+ | ``\b`` | ASCII Backspace (BS) | | +-----------------+---------------------------------+-------+ | ``\f`` | ASCII Formfeed (FF) | | +-----------------+---------------------------------+-------+ | ``\n`` | ASCII Linefeed (LF) | | +-----------------+---------------------------------+-------+ | ``\r`` | ASCII Carriage Return (CR) | | +-----------------+---------------------------------+-------+ | ``\t`` | ASCII Horizontal Tab (TAB) | | +-----------------+---------------------------------+-------+ | ``\v`` | ASCII Vertical Tab (VT) | | +-----------------+---------------------------------+-------+ | ``\ooo`` | Character with octal value | (1,3) | | | *ooo* | | +-----------------+---------------------------------+-------+ | ``\xhh`` | Character with hex value *hh* | (2,3) | +-----------------+---------------------------------+-------+ Escape sequences only recognized in string literals are: +-----------------+---------------------------------+-------+ | Escape Sequence | Meaning | Notes | +=================+=================================+=======+ | ``\N{name}`` | Character named *name* in the | \(4) | | | Unicode database | | +-----------------+---------------------------------+-------+ | ``\uxxxx`` | Character with 16-bit hex value | \(5) | | | *xxxx* | | +-----------------+---------------------------------+-------+ | ``\Uxxxxxxxx`` | Character with 32-bit hex value | \(6) | | | *xxxxxxxx* | | +-----------------+---------------------------------+-------+ Notes: (1) As in Standard C, up to three octal digits are accepted. (2) Unlike in Standard C, exactly two hex digits are required. (3) In a bytes literal, hexadecimal and octal escapes denote the byte with the given value. In a string literal, these escapes denote a Unicode character with the given value. (4) .. versionchanged:: 3.3 Support for name aliases [#]_ has been added. (5) Exactly four hex digits are required. (6) Any Unicode character can be encoded this way. Exactly eight hex digits are required. .. index:: unrecognized escape sequence Unlike Standard C, all unrecognized escape sequences are left in the string unchanged, i.e., *the backslash is left in the result*. (This behavior is useful when debugging: if an escape sequence is mistyped, the resulting output is more easily recognized as broken.) It is also important to note that the escape sequences only recognized in string literals fall into the category of unrecognized escapes for bytes literals. .. versionchanged:: 3.6 Unrecognized escape sequences produce a DeprecationWarning. In some future version of Python they will be a SyntaxError. Even in a raw literal, quotes can be escaped with a backslash, but the backslash remains in the result; for example, ``r"\""`` is a valid string literal consisting of two characters: a backslash and a double quote; ``r"\"`` is not a valid string literal (even a raw string cannot end in an odd number of backslashes). Specifically, *a raw literal cannot end in a single backslash* (since the backslash would escape the following quote character). Note also that a single backslash followed by a newline is interpreted as those two characters as part of the literal, *not* as a line continuation. .. _string-concatenation: String literal concatenation ---------------------------- Multiple adjacent string or bytes literals (delimited by whitespace), possibly using different quoting conventions, are allowed, and their meaning is the same as their concatenation. Thus, ``"hello" 'world'`` is equivalent to ``"helloworld"``. This feature can be used to reduce the number of backslashes needed, to split long strings conveniently across long lines, or even to add comments to parts of strings, for example:: re.compile("[A-Za-z_]" # letter or underscore "[A-Za-z0-9_]*" # letter, digit or underscore ) Note that this feature is defined at the syntactical level, but implemented at compile time. The '+' operator must be used to concatenate string expressions at run time. Also note that literal concatenation can use different quoting styles for each component (even mixing raw strings and triple quoted strings), and formatted string literals may be concatenated with plain string literals. .. index:: single: formatted string literal single: interpolated string literal single: string; formatted literal single: string; interpolated literal single: f-string .. _f-strings: Formatted string literals ------------------------- .. versionadded:: 3.6 A :dfn:`formatted string literal` or :dfn:`f-string` is a string literal that is prefixed with ``'f'`` or ``'F'``. These strings may contain replacement fields, which are expressions delimited by curly braces ``{}``. While other string literals always have a constant value, formatted strings are really expressions evaluated at run time. Escape sequences are decoded like in ordinary string literals (except when a literal is also marked as a raw string). After decoding, the grammar for the contents of the string is: .. productionlist:: f_string: (`literal_char` | "{{" | "}}" | `replacement_field`)* replacement_field: "{" `f_expression` ["!" `conversion`] [":" `format_spec`] "}" f_expression: (`conditional_expression` | "*" `or_expr`) : ("," `conditional_expression` | "," "*" `or_expr`)* [","] : | `yield_expression` conversion: "s" | "r" | "a" format_spec: (`literal_char` | NULL | `replacement_field`)* literal_char: The parts of the string outside curly braces are treated literally, except that any doubled curly braces ``'{{'`` or ``'}}'`` are replaced with the corresponding single curly brace. A single opening curly bracket ``'{'`` marks a replacement field, which starts with a Python expression. After the expression, there may be a conversion field, introduced by an exclamation point ``'!'``. A format specifier may also be appended, introduced by a colon ``':'``. A replacement field ends with a closing curly bracket ``'}'``. Expressions in formatted string literals are treated like regular Python expressions surrounded by parentheses, with a few exceptions. An empty expression is not allowed, and a :keyword:`lambda` expression must be surrounded by explicit parentheses. Replacement expressions can contain line breaks (e.g. in triple-quoted strings), but they cannot contain comments. Each expression is evaluated in the context where the formatted string literal appears, in order from left to right. If a conversion is specified, the result of evaluating the expression is converted before formatting. Conversion ``'!s'`` calls :func:`str` on the result, ``'!r'`` calls :func:`repr`, and ``'!a'`` calls :func:`ascii`. The result is then formatted using the :func:`format` protocol. The format specifier is passed to the :meth:`__format__` method of the expression or conversion result. An empty string is passed when the format specifier is omitted. The formatted result is then included in the final value of the whole string. Top-level format specifiers may include nested replacement fields. These nested fields may include their own conversion fields and :ref:`format specifiers `, but may not include more deeply-nested replacement fields. The :ref:`format specifier mini-language ` is the same as that used by the string .format() method. Formatted string literals may be concatenated, but replacement fields cannot be split across literals. Some examples of formatted string literals:: >>> name = "Fred" >>> f"He said his name is {name!r}." "He said his name is 'Fred'." >>> f"He said his name is {repr(name)}." # repr() is equivalent to !r "He said his name is 'Fred'." >>> width = 10 >>> precision = 4 >>> value = decimal.Decimal("12.34567") >>> f"result: {value:{width}.{precision}}" # nested fields 'result: 12.35' >>> today = datetime(year=2017, month=1, day=27) >>> f"{today:%B %d, %Y}" # using date format specifier 'January 27, 2017' >>> number = 1024 >>> f"{number:#0x}" # using integer format specifier '0x400' A consequence of sharing the same syntax as regular string literals is that characters in the replacement fields must not conflict with the quoting used in the outer formatted string literal:: f"abc {a["x"]} def" # error: outer string literal ended prematurely f"abc {a['x']} def" # workaround: use different quoting Backslashes are not allowed in format expressions and will raise an error:: f"newline: {ord('\n')}" # raises SyntaxError To include a value in which a backslash escape is required, create a temporary variable. >>> newline = ord('\n') >>> f"newline: {newline}" 'newline: 10' Formatted string literals cannot be used as docstrings, even if they do not include expressions. :: >>> def foo(): ... f"Not a docstring" ... >>> foo.__doc__ is None True See also :pep:`498` for the proposal that added formatted string literals, and :meth:`str.format`, which uses a related format string mechanism. .. _numbers: Numeric literals ---------------- .. index:: number, numeric literal, integer literal floating point literal, hexadecimal literal octal literal, binary literal, decimal literal, imaginary literal, complex literal There are three types of numeric literals: integers, floating point numbers, and imaginary numbers. There are no complex literals (complex numbers can be formed by adding a real number and an imaginary number). Note that numeric literals do not include a sign; a phrase like ``-1`` is actually an expression composed of the unary operator '``-``' and the literal ``1``. .. _integers: Integer literals ---------------- Integer literals are described by the following lexical definitions: .. productionlist:: integer: `decinteger` | `bininteger` | `octinteger` | `hexinteger` decinteger: `nonzerodigit` (["_"] `digit`)* | "0"+ (["_"] "0")* bininteger: "0" ("b" | "B") (["_"] `bindigit`)+ octinteger: "0" ("o" | "O") (["_"] `octdigit`)+ hexinteger: "0" ("x" | "X") (["_"] `hexdigit`)+ nonzerodigit: "1"..."9" digit: "0"..."9" bindigit: "0" | "1" octdigit: "0"..."7" hexdigit: `digit` | "a"..."f" | "A"..."F" There is no limit for the length of integer literals apart from what can be stored in available memory. Underscores are ignored for determining the numeric value of the literal. They can be used to group digits for enhanced readability. One underscore can occur between digits, and after base specifiers like ``0x``. Note that leading zeros in a non-zero decimal number are not allowed. This is for disambiguation with C-style octal literals, which Python used before version 3.0. Some examples of integer literals:: 7 2147483647 0o177 0b100110111 3 79228162514264337593543950336 0o377 0xdeadbeef 100_000_000_000 0b_1110_0101 .. versionchanged:: 3.6 Underscores are now allowed for grouping purposes in literals. .. _floating: Floating point literals ----------------------- Floating point literals are described by the following lexical definitions: .. productionlist:: floatnumber: `pointfloat` | `exponentfloat` pointfloat: [`digitpart`] `fraction` | `digitpart` "." exponentfloat: (`digitpart` | `pointfloat`) `exponent` digitpart: `digit` (["_"] `digit`)* fraction: "." `digitpart` exponent: ("e" | "E") ["+" | "-"] `digitpart` Note that the integer and exponent parts are always interpreted using radix 10. For example, ``077e010`` is legal, and denotes the same number as ``77e10``. The allowed range of floating point literals is implementation-dependent. As in integer literals, underscores are supported for digit grouping. Some examples of floating point literals:: 3.14 10. .001 1e100 3.14e-10 0e0 3.14_15_93 .. versionchanged:: 3.6 Underscores are now allowed for grouping purposes in literals. .. _imaginary: Imaginary literals ------------------ Imaginary literals are described by the following lexical definitions: .. productionlist:: imagnumber: (`floatnumber` | `digitpart`) ("j" | "J") An imaginary literal yields a complex number with a real part of 0.0. Complex numbers are represented as a pair of floating point numbers and have the same restrictions on their range. To create a complex number with a nonzero real part, add a floating point number to it, e.g., ``(3+4j)``. Some examples of imaginary literals:: 3.14j 10.j 10j .001j 1e100j 3.14e-10j 3.14_15_93j .. _operators: Operators ========= .. index:: single: operators The following tokens are operators: .. code-block:: none + - * ** / // % @ << >> & | ^ ~ < > <= >= == != .. _delimiters: Delimiters ========== .. index:: single: delimiters The following tokens serve as delimiters in the grammar: .. code-block:: none ( ) [ ] { } , : . ; @ = -> += -= *= /= //= %= @= &= |= ^= >>= <<= **= The period can also occur in floating-point and imaginary literals. A sequence of three periods has a special meaning as an ellipsis literal. The second half of the list, the augmented assignment operators, serve lexically as delimiters, but also perform an operation. The following printing ASCII characters have special meaning as part of other tokens or are otherwise significant to the lexical analyzer: .. code-block:: none ' " # \ The following printing ASCII characters are not used in Python. Their occurrence outside string literals and comments is an unconditional error: .. code-block:: none $ ? ` .. rubric:: Footnotes .. [#] http://www.unicode.org/Public/9.0.0/ucd/NameAliases.txt PK 3],p&LLreference/expressions.rst.txtnu[ .. _expressions: *********** Expressions *********** .. index:: expression, BNF This chapter explains the meaning of the elements of expressions in Python. **Syntax Notes:** In this and the following chapters, extended BNF notation will be used to describe syntax, not lexical analysis. When (one alternative of) a syntax rule has the form .. productionlist:: * name: `othername` and no semantics are given, the semantics of this form of ``name`` are the same as for ``othername``. .. _conversions: Arithmetic conversions ====================== .. index:: pair: arithmetic; conversion When a description of an arithmetic operator below uses the phrase "the numeric arguments are converted to a common type," this means that the operator implementation for built-in types works as follows: * If either argument is a complex number, the other is converted to complex; * otherwise, if either argument is a floating point number, the other is converted to floating point; * otherwise, both must be integers and no conversion is necessary. Some additional rules apply for certain operators (e.g., a string as a left argument to the '%' operator). Extensions must define their own conversion behavior. .. _atoms: Atoms ===== .. index:: atom Atoms are the most basic elements of expressions. The simplest atoms are identifiers or literals. Forms enclosed in parentheses, brackets or braces are also categorized syntactically as atoms. The syntax for atoms is: .. productionlist:: atom: `identifier` | `literal` | `enclosure` enclosure: `parenth_form` | `list_display` | `dict_display` | `set_display` : | `generator_expression` | `yield_atom` .. _atom-identifiers: Identifiers (Names) ------------------- .. index:: name, identifier An identifier occurring as an atom is a name. See section :ref:`identifiers` for lexical definition and section :ref:`naming` for documentation of naming and binding. .. index:: exception: NameError When the name is bound to an object, evaluation of the atom yields that object. When a name is not bound, an attempt to evaluate it raises a :exc:`NameError` exception. .. index:: pair: name; mangling pair: private; names **Private name mangling:** When an identifier that textually occurs in a class definition begins with two or more underscore characters and does not end in two or more underscores, it is considered a :dfn:`private name` of that class. Private names are transformed to a longer form before code is generated for them. The transformation inserts the class name, with leading underscores removed and a single underscore inserted, in front of the name. For example, the identifier ``__spam`` occurring in a class named ``Ham`` will be transformed to ``_Ham__spam``. This transformation is independent of the syntactical context in which the identifier is used. If the transformed name is extremely long (longer than 255 characters), implementation defined truncation may happen. If the class name consists only of underscores, no transformation is done. .. _atom-literals: Literals -------- .. index:: single: literal Python supports string and bytes literals and various numeric literals: .. productionlist:: literal: `stringliteral` | `bytesliteral` : | `integer` | `floatnumber` | `imagnumber` Evaluation of a literal yields an object of the given type (string, bytes, integer, floating point number, complex number) with the given value. The value may be approximated in the case of floating point and imaginary (complex) literals. See section :ref:`literals` for details. .. index:: triple: immutable; data; type pair: immutable; object All literals correspond to immutable data types, and hence the object's identity is less important than its value. Multiple evaluations of literals with the same value (either the same occurrence in the program text or a different occurrence) may obtain the same object or a different object with the same value. .. _parenthesized: Parenthesized forms ------------------- .. index:: single: parenthesized form A parenthesized form is an optional expression list enclosed in parentheses: .. productionlist:: parenth_form: "(" [`starred_expression`] ")" A parenthesized expression list yields whatever that expression list yields: if the list contains at least one comma, it yields a tuple; otherwise, it yields the single expression that makes up the expression list. .. index:: pair: empty; tuple An empty pair of parentheses yields an empty tuple object. Since tuples are immutable, the rules for literals apply (i.e., two occurrences of the empty tuple may or may not yield the same object). .. index:: single: comma pair: tuple; display Note that tuples are not formed by the parentheses, but rather by use of the comma operator. The exception is the empty tuple, for which parentheses *are* required --- allowing unparenthesized "nothing" in expressions would cause ambiguities and allow common typos to pass uncaught. .. _comprehensions: Displays for lists, sets and dictionaries ----------------------------------------- For constructing a list, a set or a dictionary Python provides special syntax called "displays", each of them in two flavors: * either the container contents are listed explicitly, or * they are computed via a set of looping and filtering instructions, called a :dfn:`comprehension`. Common syntax elements for comprehensions are: .. productionlist:: comprehension: `expression` `comp_for` comp_for: [ASYNC] "for" `target_list` "in" `or_test` [`comp_iter`] comp_iter: `comp_for` | `comp_if` comp_if: "if" `expression_nocond` [`comp_iter`] The comprehension consists of a single expression followed by at least one :keyword:`for` clause and zero or more :keyword:`for` or :keyword:`if` clauses. In this case, the elements of the new container are those that would be produced by considering each of the :keyword:`for` or :keyword:`if` clauses a block, nesting from left to right, and evaluating the expression to produce an element each time the innermost block is reached. Note that the comprehension is executed in a separate scope, so names assigned to in the target list don't "leak" into the enclosing scope. Since Python 3.6, in an :keyword:`async def` function, an :keyword:`async for` clause may be used to iterate over a :term:`asynchronous iterator`. A comprehension in an :keyword:`async def` function may consist of either a :keyword:`for` or :keyword:`async for` clause following the leading expression, may contain additional :keyword:`for` or :keyword:`async for` clauses, and may also use :keyword:`await` expressions. If a comprehension contains either :keyword:`async for` clauses or :keyword:`await` expressions it is called an :dfn:`asynchronous comprehension`. An asynchronous comprehension may suspend the execution of the coroutine function in which it appears. See also :pep:`530`. .. _lists: List displays ------------- .. index:: pair: list; display pair: list; comprehensions pair: empty; list object: list A list display is a possibly empty series of expressions enclosed in square brackets: .. productionlist:: list_display: "[" [`starred_list` | `comprehension`] "]" A list display yields a new list object, the contents being specified by either a list of expressions or a comprehension. When a comma-separated list of expressions is supplied, its elements are evaluated from left to right and placed into the list object in that order. When a comprehension is supplied, the list is constructed from the elements resulting from the comprehension. .. _set: Set displays ------------ .. index:: pair: set; display object: set A set display is denoted by curly braces and distinguishable from dictionary displays by the lack of colons separating keys and values: .. productionlist:: set_display: "{" (`starred_list` | `comprehension`) "}" A set display yields a new mutable set object, the contents being specified by either a sequence of expressions or a comprehension. When a comma-separated list of expressions is supplied, its elements are evaluated from left to right and added to the set object. When a comprehension is supplied, the set is constructed from the elements resulting from the comprehension. An empty set cannot be constructed with ``{}``; this literal constructs an empty dictionary. .. _dict: Dictionary displays ------------------- .. index:: pair: dictionary; display key, datum, key/datum pair object: dictionary A dictionary display is a possibly empty series of key/datum pairs enclosed in curly braces: .. productionlist:: dict_display: "{" [`key_datum_list` | `dict_comprehension`] "}" key_datum_list: `key_datum` ("," `key_datum`)* [","] key_datum: `expression` ":" `expression` | "**" `or_expr` dict_comprehension: `expression` ":" `expression` `comp_for` A dictionary display yields a new dictionary object. If a comma-separated sequence of key/datum pairs is given, they are evaluated from left to right to define the entries of the dictionary: each key object is used as a key into the dictionary to store the corresponding datum. This means that you can specify the same key multiple times in the key/datum list, and the final dictionary's value for that key will be the last one given. .. index:: unpacking; dictionary, **; in dictionary displays A double asterisk ``**`` denotes :dfn:`dictionary unpacking`. Its operand must be a :term:`mapping`. Each mapping item is added to the new dictionary. Later values replace values already set by earlier key/datum pairs and earlier dictionary unpackings. .. versionadded:: 3.5 Unpacking into dictionary displays, originally proposed by :pep:`448`. A dict comprehension, in contrast to list and set comprehensions, needs two expressions separated with a colon followed by the usual "for" and "if" clauses. When the comprehension is run, the resulting key and value elements are inserted in the new dictionary in the order they are produced. .. index:: pair: immutable; object hashable Restrictions on the types of the key values are listed earlier in section :ref:`types`. (To summarize, the key type should be :term:`hashable`, which excludes all mutable objects.) Clashes between duplicate keys are not detected; the last datum (textually rightmost in the display) stored for a given key value prevails. .. _genexpr: Generator expressions --------------------- .. index:: pair: generator; expression object: generator A generator expression is a compact generator notation in parentheses: .. productionlist:: generator_expression: "(" `expression` `comp_for` ")" A generator expression yields a new generator object. Its syntax is the same as for comprehensions, except that it is enclosed in parentheses instead of brackets or curly braces. Variables used in the generator expression are evaluated lazily when the :meth:`~generator.__next__` method is called for the generator object (in the same fashion as normal generators). However, the leftmost :keyword:`for` clause is immediately evaluated, so that an error produced by it can be seen before any other possible error in the code that handles the generator expression. Subsequent :keyword:`for` clauses cannot be evaluated immediately since they may depend on the previous :keyword:`for` loop. For example: ``(x*y for x in range(10) for y in bar(x))``. The parentheses can be omitted on calls with only one argument. See section :ref:`calls` for details. Since Python 3.6, if the generator appears in an :keyword:`async def` function, then :keyword:`async for` clauses and :keyword:`await` expressions are permitted as with an asynchronous comprehension. If a generator expression contains either :keyword:`async for` clauses or :keyword:`await` expressions it is called an :dfn:`asynchronous generator expression`. An asynchronous generator expression yields a new asynchronous generator object, which is an asynchronous iterator (see :ref:`async-iterators`). .. _yieldexpr: Yield expressions ----------------- .. index:: keyword: yield pair: yield; expression pair: generator; function .. productionlist:: yield_atom: "(" `yield_expression` ")" yield_expression: "yield" [`expression_list` | "from" `expression`] The yield expression is used when defining a :term:`generator` function or an :term:`asynchronous generator` function and thus can only be used in the body of a function definition. Using a yield expression in a function's body causes that function to be a generator, and using it in an :keyword:`async def` function's body causes that coroutine function to be an asynchronous generator. For example:: def gen(): # defines a generator function yield 123 async def agen(): # defines an asynchronous generator function (PEP 525) yield 123 Generator functions are described below, while asynchronous generator functions are described separately in section :ref:`asynchronous-generator-functions`. When a generator function is called, it returns an iterator known as a generator. That generator then controls the execution of the generator function. The execution starts when one of the generator's methods is called. At that time, the execution proceeds to the first yield expression, where it is suspended again, returning the value of :token:`expression_list` to the generator's caller. By suspended, we mean that all local state is retained, including the current bindings of local variables, the instruction pointer, the internal evaluation stack, and the state of any exception handling. When the execution is resumed by calling one of the generator's methods, the function can proceed exactly as if the yield expression were just another external call. The value of the yield expression after resuming depends on the method which resumed the execution. If :meth:`~generator.__next__` is used (typically via either a :keyword:`for` or the :func:`next` builtin) then the result is :const:`None`. Otherwise, if :meth:`~generator.send` is used, then the result will be the value passed in to that method. .. index:: single: coroutine All of this makes generator functions quite similar to coroutines; they yield multiple times, they have more than one entry point and their execution can be suspended. The only difference is that a generator function cannot control where the execution should continue after it yields; the control is always transferred to the generator's caller. Yield expressions are allowed anywhere in a :keyword:`try` construct. If the generator is not resumed before it is finalized (by reaching a zero reference count or by being garbage collected), the generator-iterator's :meth:`~generator.close` method will be called, allowing any pending :keyword:`finally` clauses to execute. When ``yield from `` is used, it treats the supplied expression as a subiterator. All values produced by that subiterator are passed directly to the caller of the current generator's methods. Any values passed in with :meth:`~generator.send` and any exceptions passed in with :meth:`~generator.throw` are passed to the underlying iterator if it has the appropriate methods. If this is not the case, then :meth:`~generator.send` will raise :exc:`AttributeError` or :exc:`TypeError`, while :meth:`~generator.throw` will just raise the passed in exception immediately. When the underlying iterator is complete, the :attr:`~StopIteration.value` attribute of the raised :exc:`StopIteration` instance becomes the value of the yield expression. It can be either set explicitly when raising :exc:`StopIteration`, or automatically when the sub-iterator is a generator (by returning a value from the sub-generator). .. versionchanged:: 3.3 Added ``yield from `` to delegate control flow to a subiterator. The parentheses may be omitted when the yield expression is the sole expression on the right hand side of an assignment statement. .. seealso:: :pep:`255` - Simple Generators The proposal for adding generators and the :keyword:`yield` statement to Python. :pep:`342` - Coroutines via Enhanced Generators The proposal to enhance the API and syntax of generators, making them usable as simple coroutines. :pep:`380` - Syntax for Delegating to a Subgenerator The proposal to introduce the :token:`yield_from` syntax, making delegation to sub-generators easy. .. index:: object: generator .. _generator-methods: Generator-iterator methods ^^^^^^^^^^^^^^^^^^^^^^^^^^ This subsection describes the methods of a generator iterator. They can be used to control the execution of a generator function. Note that calling any of the generator methods below when the generator is already executing raises a :exc:`ValueError` exception. .. index:: exception: StopIteration .. method:: generator.__next__() Starts the execution of a generator function or resumes it at the last executed yield expression. When a generator function is resumed with a :meth:`~generator.__next__` method, the current yield expression always evaluates to :const:`None`. The execution then continues to the next yield expression, where the generator is suspended again, and the value of the :token:`expression_list` is returned to :meth:`__next__`'s caller. If the generator exits without yielding another value, a :exc:`StopIteration` exception is raised. This method is normally called implicitly, e.g. by a :keyword:`for` loop, or by the built-in :func:`next` function. .. method:: generator.send(value) Resumes the execution and "sends" a value into the generator function. The *value* argument becomes the result of the current yield expression. The :meth:`send` method returns the next value yielded by the generator, or raises :exc:`StopIteration` if the generator exits without yielding another value. When :meth:`send` is called to start the generator, it must be called with :const:`None` as the argument, because there is no yield expression that could receive the value. .. method:: generator.throw(type[, value[, traceback]]) Raises an exception of type ``type`` at the point where the generator was paused, and returns the next value yielded by the generator function. If the generator exits without yielding another value, a :exc:`StopIteration` exception is raised. If the generator function does not catch the passed-in exception, or raises a different exception, then that exception propagates to the caller. .. index:: exception: GeneratorExit .. method:: generator.close() Raises a :exc:`GeneratorExit` at the point where the generator function was paused. If the generator function then exits gracefully, is already closed, or raises :exc:`GeneratorExit` (by not catching the exception), close returns to its caller. If the generator yields a value, a :exc:`RuntimeError` is raised. If the generator raises any other exception, it is propagated to the caller. :meth:`close` does nothing if the generator has already exited due to an exception or normal exit. .. index:: single: yield; examples Examples ^^^^^^^^ Here is a simple example that demonstrates the behavior of generators and generator functions:: >>> def echo(value=None): ... print("Execution starts when 'next()' is called for the first time.") ... try: ... while True: ... try: ... value = (yield value) ... except Exception as e: ... value = e ... finally: ... print("Don't forget to clean up when 'close()' is called.") ... >>> generator = echo(1) >>> print(next(generator)) Execution starts when 'next()' is called for the first time. 1 >>> print(next(generator)) None >>> print(generator.send(2)) 2 >>> generator.throw(TypeError, "spam") TypeError('spam',) >>> generator.close() Don't forget to clean up when 'close()' is called. For examples using ``yield from``, see :ref:`pep-380` in "What's New in Python." .. _asynchronous-generator-functions: Asynchronous generator functions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The presence of a yield expression in a function or method defined using :keyword:`async def` further defines the function as a :term:`asynchronous generator` function. When an asynchronous generator function is called, it returns an asynchronous iterator known as an asynchronous generator object. That object then controls the execution of the generator function. An asynchronous generator object is typically used in an :keyword:`async for` statement in a coroutine function analogously to how a generator object would be used in a :keyword:`for` statement. Calling one of the asynchronous generator's methods returns an :term:`awaitable` object, and the execution starts when this object is awaited on. At that time, the execution proceeds to the first yield expression, where it is suspended again, returning the value of :token:`expression_list` to the awaiting coroutine. As with a generator, suspension means that all local state is retained, including the current bindings of local variables, the instruction pointer, the internal evaluation stack, and the state of any exception handling. When the execution is resumed by awaiting on the next object returned by the asynchronous generator's methods, the function can proceed exactly as if the yield expression were just another external call. The value of the yield expression after resuming depends on the method which resumed the execution. If :meth:`~agen.__anext__` is used then the result is :const:`None`. Otherwise, if :meth:`~agen.asend` is used, then the result will be the value passed in to that method. In an asynchronous generator function, yield expressions are allowed anywhere in a :keyword:`try` construct. However, if an asynchronous generator is not resumed before it is finalized (by reaching a zero reference count or by being garbage collected), then a yield expression within a :keyword:`try` construct could result in a failure to execute pending :keyword:`finally` clauses. In this case, it is the responsibility of the event loop or scheduler running the asynchronous generator to call the asynchronous generator-iterator's :meth:`~agen.aclose` method and run the resulting coroutine object, thus allowing any pending :keyword:`finally` clauses to execute. To take care of finalization, an event loop should define a *finalizer* function which takes an asynchronous generator-iterator and presumably calls :meth:`~agen.aclose` and executes the coroutine. This *finalizer* may be registered by calling :func:`sys.set_asyncgen_hooks`. When first iterated over, an asynchronous generator-iterator will store the registered *finalizer* to be called upon finalization. For a reference example of a *finalizer* method see the implementation of ``asyncio.Loop.shutdown_asyncgens`` in :source:`Lib/asyncio/base_events.py`. The expression ``yield from `` is a syntax error when used in an asynchronous generator function. .. index:: object: asynchronous-generator .. _asynchronous-generator-methods: Asynchronous generator-iterator methods ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This subsection describes the methods of an asynchronous generator iterator, which are used to control the execution of a generator function. .. index:: exception: StopAsyncIteration .. coroutinemethod:: agen.__anext__() Returns an awaitable which when run starts to execute the asynchronous generator or resumes it at the last executed yield expression. When an asynchronous generator function is resumed with a :meth:`~agen.__anext__` method, the current yield expression always evaluates to :const:`None` in the returned awaitable, which when run will continue to the next yield expression. The value of the :token:`expression_list` of the yield expression is the value of the :exc:`StopIteration` exception raised by the completing coroutine. If the asynchronous generator exits without yielding another value, the awaitable instead raises an :exc:`StopAsyncIteration` exception, signalling that the asynchronous iteration has completed. This method is normally called implicitly by a :keyword:`async for` loop. .. coroutinemethod:: agen.asend(value) Returns an awaitable which when run resumes the execution of the asynchronous generator. As with the :meth:`~generator.send()` method for a generator, this "sends" a value into the asynchronous generator function, and the *value* argument becomes the result of the current yield expression. The awaitable returned by the :meth:`asend` method will return the next value yielded by the generator as the value of the raised :exc:`StopIteration`, or raises :exc:`StopAsyncIteration` if the asynchronous generator exits without yielding another value. When :meth:`asend` is called to start the asynchronous generator, it must be called with :const:`None` as the argument, because there is no yield expression that could receive the value. .. coroutinemethod:: agen.athrow(type[, value[, traceback]]) Returns an awaitable that raises an exception of type ``type`` at the point where the asynchronous generator was paused, and returns the next value yielded by the generator function as the value of the raised :exc:`StopIteration` exception. If the asynchronous generator exits without yielding another value, an :exc:`StopAsyncIteration` exception is raised by the awaitable. If the generator function does not catch the passed-in exception, or raises a different exception, then when the awaitable is run that exception propagates to the caller of the awaitable. .. index:: exception: GeneratorExit .. coroutinemethod:: agen.aclose() Returns an awaitable that when run will throw a :exc:`GeneratorExit` into the asynchronous generator function at the point where it was paused. If the asynchronous generator function then exits gracefully, is already closed, or raises :exc:`GeneratorExit` (by not catching the exception), then the returned awaitable will raise a :exc:`StopIteration` exception. Any further awaitables returned by subsequent calls to the asynchronous generator will raise a :exc:`StopAsyncIteration` exception. If the asynchronous generator yields a value, a :exc:`RuntimeError` is raised by the awaitable. If the asynchronous generator raises any other exception, it is propagated to the caller of the awaitable. If the asynchronous generator has already exited due to an exception or normal exit, then further calls to :meth:`aclose` will return an awaitable that does nothing. .. _primaries: Primaries ========= .. index:: single: primary Primaries represent the most tightly bound operations of the language. Their syntax is: .. productionlist:: primary: `atom` | `attributeref` | `subscription` | `slicing` | `call` .. _attribute-references: Attribute references -------------------- .. index:: pair: attribute; reference An attribute reference is a primary followed by a period and a name: .. productionlist:: attributeref: `primary` "." `identifier` .. index:: exception: AttributeError object: module object: list The primary must evaluate to an object of a type that supports attribute references, which most objects do. This object is then asked to produce the attribute whose name is the identifier. This production can be customized by overriding the :meth:`__getattr__` method. If this attribute is not available, the exception :exc:`AttributeError` is raised. Otherwise, the type and value of the object produced is determined by the object. Multiple evaluations of the same attribute reference may yield different objects. .. _subscriptions: Subscriptions ------------- .. index:: single: subscription .. index:: object: sequence object: mapping object: string object: tuple object: list object: dictionary pair: sequence; item A subscription selects an item of a sequence (string, tuple or list) or mapping (dictionary) object: .. productionlist:: subscription: `primary` "[" `expression_list` "]" The primary must evaluate to an object that supports subscription (lists or dictionaries for example). User-defined objects can support subscription by defining a :meth:`__getitem__` method. For built-in objects, there are two types of objects that support subscription: If the primary is a mapping, the expression list must evaluate to an object whose value is one of the keys of the mapping, and the subscription selects the value in the mapping that corresponds to that key. (The expression list is a tuple except if it has exactly one item.) If the primary is a sequence, the expression list must evaluate to an integer or a slice (as discussed in the following section). The formal syntax makes no special provision for negative indices in sequences; however, built-in sequences all provide a :meth:`__getitem__` method that interprets negative indices by adding the length of the sequence to the index (so that ``x[-1]`` selects the last item of ``x``). The resulting value must be a nonnegative integer less than the number of items in the sequence, and the subscription selects the item whose index is that value (counting from zero). Since the support for negative indices and slicing occurs in the object's :meth:`__getitem__` method, subclasses overriding this method will need to explicitly add that support. .. index:: single: character pair: string; item A string's items are characters. A character is not a separate data type but a string of exactly one character. .. _slicings: Slicings -------- .. index:: single: slicing single: slice .. index:: object: sequence object: string object: tuple object: list A slicing selects a range of items in a sequence object (e.g., a string, tuple or list). Slicings may be used as expressions or as targets in assignment or :keyword:`del` statements. The syntax for a slicing: .. productionlist:: slicing: `primary` "[" `slice_list` "]" slice_list: `slice_item` ("," `slice_item`)* [","] slice_item: `expression` | `proper_slice` proper_slice: [`lower_bound`] ":" [`upper_bound`] [ ":" [`stride`] ] lower_bound: `expression` upper_bound: `expression` stride: `expression` There is ambiguity in the formal syntax here: anything that looks like an expression list also looks like a slice list, so any subscription can be interpreted as a slicing. Rather than further complicating the syntax, this is disambiguated by defining that in this case the interpretation as a subscription takes priority over the interpretation as a slicing (this is the case if the slice list contains no proper slice). .. index:: single: start (slice object attribute) single: stop (slice object attribute) single: step (slice object attribute) The semantics for a slicing are as follows. The primary is indexed (using the same :meth:`__getitem__` method as normal subscription) with a key that is constructed from the slice list, as follows. If the slice list contains at least one comma, the key is a tuple containing the conversion of the slice items; otherwise, the conversion of the lone slice item is the key. The conversion of a slice item that is an expression is that expression. The conversion of a proper slice is a slice object (see section :ref:`types`) whose :attr:`~slice.start`, :attr:`~slice.stop` and :attr:`~slice.step` attributes are the values of the expressions given as lower bound, upper bound and stride, respectively, substituting ``None`` for missing expressions. .. index:: object: callable single: call single: argument; call semantics .. _calls: Calls ----- A call calls a callable object (e.g., a :term:`function`) with a possibly empty series of :term:`arguments `: .. productionlist:: call: `primary` "(" [`argument_list` [","] | `comprehension`] ")" argument_list: `positional_arguments` ["," `starred_and_keywords`] : ["," `keywords_arguments`] : | `starred_and_keywords` ["," `keywords_arguments`] : | `keywords_arguments` positional_arguments: ["*"] `expression` ("," ["*"] `expression`)* starred_and_keywords: ("*" `expression` | `keyword_item`) : ("," "*" `expression` | "," `keyword_item`)* keywords_arguments: (`keyword_item` | "**" `expression`) : ("," `keyword_item` | "," "**" `expression`)* keyword_item: `identifier` "=" `expression` An optional trailing comma may be present after the positional and keyword arguments but does not affect the semantics. .. index:: single: parameter; call semantics The primary must evaluate to a callable object (user-defined functions, built-in functions, methods of built-in objects, class objects, methods of class instances, and all objects having a :meth:`__call__` method are callable). All argument expressions are evaluated before the call is attempted. Please refer to section :ref:`function` for the syntax of formal :term:`parameter` lists. .. XXX update with kwonly args PEP If keyword arguments are present, they are first converted to positional arguments, as follows. First, a list of unfilled slots is created for the formal parameters. If there are N positional arguments, they are placed in the first N slots. Next, for each keyword argument, the identifier is used to determine the corresponding slot (if the identifier is the same as the first formal parameter name, the first slot is used, and so on). If the slot is already filled, a :exc:`TypeError` exception is raised. Otherwise, the value of the argument is placed in the slot, filling it (even if the expression is ``None``, it fills the slot). When all arguments have been processed, the slots that are still unfilled are filled with the corresponding default value from the function definition. (Default values are calculated, once, when the function is defined; thus, a mutable object such as a list or dictionary used as default value will be shared by all calls that don't specify an argument value for the corresponding slot; this should usually be avoided.) If there are any unfilled slots for which no default value is specified, a :exc:`TypeError` exception is raised. Otherwise, the list of filled slots is used as the argument list for the call. .. impl-detail:: An implementation may provide built-in functions whose positional parameters do not have names, even if they are 'named' for the purpose of documentation, and which therefore cannot be supplied by keyword. In CPython, this is the case for functions implemented in C that use :c:func:`PyArg_ParseTuple` to parse their arguments. If there are more positional arguments than there are formal parameter slots, a :exc:`TypeError` exception is raised, unless a formal parameter using the syntax ``*identifier`` is present; in this case, that formal parameter receives a tuple containing the excess positional arguments (or an empty tuple if there were no excess positional arguments). If any keyword argument does not correspond to a formal parameter name, a :exc:`TypeError` exception is raised, unless a formal parameter using the syntax ``**identifier`` is present; in this case, that formal parameter receives a dictionary containing the excess keyword arguments (using the keywords as keys and the argument values as corresponding values), or a (new) empty dictionary if there were no excess keyword arguments. .. index:: single: *; in function calls single: unpacking; in function calls If the syntax ``*expression`` appears in the function call, ``expression`` must evaluate to an :term:`iterable`. Elements from these iterables are treated as if they were additional positional arguments. For the call ``f(x1, x2, *y, x3, x4)``, if *y* evaluates to a sequence *y1*, ..., *yM*, this is equivalent to a call with M+4 positional arguments *x1*, *x2*, *y1*, ..., *yM*, *x3*, *x4*. A consequence of this is that although the ``*expression`` syntax may appear *after* explicit keyword arguments, it is processed *before* the keyword arguments (and any ``**expression`` arguments -- see below). So:: >>> def f(a, b): ... print(a, b) ... >>> f(b=1, *(2,)) 2 1 >>> f(a=1, *(2,)) Traceback (most recent call last): File "", line 1, in TypeError: f() got multiple values for keyword argument 'a' >>> f(1, *(2,)) 1 2 It is unusual for both keyword arguments and the ``*expression`` syntax to be used in the same call, so in practice this confusion does not arise. .. index:: single: **; in function calls If the syntax ``**expression`` appears in the function call, ``expression`` must evaluate to a :term:`mapping`, the contents of which are treated as additional keyword arguments. If a keyword is already present (as an explicit keyword argument, or from another unpacking), a :exc:`TypeError` exception is raised. Formal parameters using the syntax ``*identifier`` or ``**identifier`` cannot be used as positional argument slots or as keyword argument names. .. versionchanged:: 3.5 Function calls accept any number of ``*`` and ``**`` unpackings, positional arguments may follow iterable unpackings (``*``), and keyword arguments may follow dictionary unpackings (``**``). Originally proposed by :pep:`448`. A call always returns some value, possibly ``None``, unless it raises an exception. How this value is computed depends on the type of the callable object. If it is--- a user-defined function: .. index:: pair: function; call triple: user-defined; function; call object: user-defined function object: function The code block for the function is executed, passing it the argument list. The first thing the code block will do is bind the formal parameters to the arguments; this is described in section :ref:`function`. When the code block executes a :keyword:`return` statement, this specifies the return value of the function call. a built-in function or method: .. index:: pair: function; call pair: built-in function; call pair: method; call pair: built-in method; call object: built-in method object: built-in function object: method object: function The result is up to the interpreter; see :ref:`built-in-funcs` for the descriptions of built-in functions and methods. a class object: .. index:: object: class pair: class object; call A new instance of that class is returned. a class instance method: .. index:: object: class instance object: instance pair: class instance; call The corresponding user-defined function is called, with an argument list that is one longer than the argument list of the call: the instance becomes the first argument. a class instance: .. index:: pair: instance; call single: __call__() (object method) The class must define a :meth:`__call__` method; the effect is then the same as if that method was called. .. _await: Await expression ================ Suspend the execution of :term:`coroutine` on an :term:`awaitable` object. Can only be used inside a :term:`coroutine function`. .. productionlist:: await_expr: "await" `primary` .. versionadded:: 3.5 .. _power: The power operator ================== The power operator binds more tightly than unary operators on its left; it binds less tightly than unary operators on its right. The syntax is: .. productionlist:: power: (`await_expr` | `primary`) ["**" `u_expr`] Thus, in an unparenthesized sequence of power and unary operators, the operators are evaluated from right to left (this does not constrain the evaluation order for the operands): ``-1**2`` results in ``-1``. The power operator has the same semantics as the built-in :func:`pow` function, when called with two arguments: it yields its left argument raised to the power of its right argument. The numeric arguments are first converted to a common type, and the result is of that type. For int operands, the result has the same type as the operands unless the second argument is negative; in that case, all arguments are converted to float and a float result is delivered. For example, ``10**2`` returns ``100``, but ``10**-2`` returns ``0.01``. Raising ``0.0`` to a negative power results in a :exc:`ZeroDivisionError`. Raising a negative number to a fractional power results in a :class:`complex` number. (In earlier versions it raised a :exc:`ValueError`.) .. _unary: Unary arithmetic and bitwise operations ======================================= .. index:: triple: unary; arithmetic; operation triple: unary; bitwise; operation All unary arithmetic and bitwise operations have the same priority: .. productionlist:: u_expr: `power` | "-" `u_expr` | "+" `u_expr` | "~" `u_expr` .. index:: single: negation single: minus The unary ``-`` (minus) operator yields the negation of its numeric argument. .. index:: single: plus The unary ``+`` (plus) operator yields its numeric argument unchanged. .. index:: single: inversion The unary ``~`` (invert) operator yields the bitwise inversion of its integer argument. The bitwise inversion of ``x`` is defined as ``-(x+1)``. It only applies to integral numbers. .. index:: exception: TypeError In all three cases, if the argument does not have the proper type, a :exc:`TypeError` exception is raised. .. _binary: Binary arithmetic operations ============================ .. index:: triple: binary; arithmetic; operation The binary arithmetic operations have the conventional priority levels. Note that some of these operations also apply to certain non-numeric types. Apart from the power operator, there are only two levels, one for multiplicative operators and one for additive operators: .. productionlist:: m_expr: `u_expr` | `m_expr` "*" `u_expr` | `m_expr` "@" `m_expr` | : `m_expr` "//" `u_expr` | `m_expr` "/" `u_expr` | : `m_expr` "%" `u_expr` a_expr: `m_expr` | `a_expr` "+" `m_expr` | `a_expr` "-" `m_expr` .. index:: single: multiplication The ``*`` (multiplication) operator yields the product of its arguments. The arguments must either both be numbers, or one argument must be an integer and the other must be a sequence. In the former case, the numbers are converted to a common type and then multiplied together. In the latter case, sequence repetition is performed; a negative repetition factor yields an empty sequence. .. index:: single: matrix multiplication operator: @ The ``@`` (at) operator is intended to be used for matrix multiplication. No builtin Python types implement this operator. .. versionadded:: 3.5 .. index:: exception: ZeroDivisionError single: division The ``/`` (division) and ``//`` (floor division) operators yield the quotient of their arguments. The numeric arguments are first converted to a common type. Division of integers yields a float, while floor division of integers results in an integer; the result is that of mathematical division with the 'floor' function applied to the result. Division by zero raises the :exc:`ZeroDivisionError` exception. .. index:: single: modulo The ``%`` (modulo) operator yields the remainder from the division of the first argument by the second. The numeric arguments are first converted to a common type. A zero right argument raises the :exc:`ZeroDivisionError` exception. The arguments may be floating point numbers, e.g., ``3.14%0.7`` equals ``0.34`` (since ``3.14`` equals ``4*0.7 + 0.34``.) The modulo operator always yields a result with the same sign as its second operand (or zero); the absolute value of the result is strictly smaller than the absolute value of the second operand [#]_. The floor division and modulo operators are connected by the following identity: ``x == (x//y)*y + (x%y)``. Floor division and modulo are also connected with the built-in function :func:`divmod`: ``divmod(x, y) == (x//y, x%y)``. [#]_. In addition to performing the modulo operation on numbers, the ``%`` operator is also overloaded by string objects to perform old-style string formatting (also known as interpolation). The syntax for string formatting is described in the Python Library Reference, section :ref:`old-string-formatting`. The floor division operator, the modulo operator, and the :func:`divmod` function are not defined for complex numbers. Instead, convert to a floating point number using the :func:`abs` function if appropriate. .. index:: single: addition The ``+`` (addition) operator yields the sum of its arguments. The arguments must either both be numbers or both be sequences of the same type. In the former case, the numbers are converted to a common type and then added together. In the latter case, the sequences are concatenated. .. index:: single: subtraction The ``-`` (subtraction) operator yields the difference of its arguments. The numeric arguments are first converted to a common type. .. _shifting: Shifting operations =================== .. index:: pair: shifting; operation The shifting operations have lower priority than the arithmetic operations: .. productionlist:: shift_expr: `a_expr` | `shift_expr` ("<<" | ">>") `a_expr` These operators accept integers as arguments. They shift the first argument to the left or right by the number of bits given by the second argument. .. index:: exception: ValueError A right shift by *n* bits is defined as floor division by ``pow(2,n)``. A left shift by *n* bits is defined as multiplication with ``pow(2,n)``. .. note:: In the current implementation, the right-hand operand is required to be at most :attr:`sys.maxsize`. If the right-hand operand is larger than :attr:`sys.maxsize` an :exc:`OverflowError` exception is raised. .. _bitwise: Binary bitwise operations ========================= .. index:: triple: binary; bitwise; operation Each of the three bitwise operations has a different priority level: .. productionlist:: and_expr: `shift_expr` | `and_expr` "&" `shift_expr` xor_expr: `and_expr` | `xor_expr` "^" `and_expr` or_expr: `xor_expr` | `or_expr` "|" `xor_expr` .. index:: pair: bitwise; and The ``&`` operator yields the bitwise AND of its arguments, which must be integers. .. index:: pair: bitwise; xor pair: exclusive; or The ``^`` operator yields the bitwise XOR (exclusive OR) of its arguments, which must be integers. .. index:: pair: bitwise; or pair: inclusive; or The ``|`` operator yields the bitwise (inclusive) OR of its arguments, which must be integers. .. _comparisons: Comparisons =========== .. index:: single: comparison .. index:: pair: C; language Unlike C, all comparison operations in Python have the same priority, which is lower than that of any arithmetic, shifting or bitwise operation. Also unlike C, expressions like ``a < b < c`` have the interpretation that is conventional in mathematics: .. productionlist:: comparison: `or_expr` (`comp_operator` `or_expr`)* comp_operator: "<" | ">" | "==" | ">=" | "<=" | "!=" : | "is" ["not"] | ["not"] "in" Comparisons yield boolean values: ``True`` or ``False``. .. index:: pair: chaining; comparisons Comparisons can be chained arbitrarily, e.g., ``x < y <= z`` is equivalent to ``x < y and y <= z``, except that ``y`` is evaluated only once (but in both cases ``z`` is not evaluated at all when ``x < y`` is found to be false). Formally, if *a*, *b*, *c*, ..., *y*, *z* are expressions and *op1*, *op2*, ..., *opN* are comparison operators, then ``a op1 b op2 c ... y opN z`` is equivalent to ``a op1 b and b op2 c and ... y opN z``, except that each expression is evaluated at most once. Note that ``a op1 b op2 c`` doesn't imply any kind of comparison between *a* and *c*, so that, e.g., ``x < y > z`` is perfectly legal (though perhaps not pretty). Value comparisons ----------------- The operators ``<``, ``>``, ``==``, ``>=``, ``<=``, and ``!=`` compare the values of two objects. The objects do not need to have the same type. Chapter :ref:`objects` states that objects have a value (in addition to type and identity). The value of an object is a rather abstract notion in Python: For example, there is no canonical access method for an object's value. Also, there is no requirement that the value of an object should be constructed in a particular way, e.g. comprised of all its data attributes. Comparison operators implement a particular notion of what the value of an object is. One can think of them as defining the value of an object indirectly, by means of their comparison implementation. Because all types are (direct or indirect) subtypes of :class:`object`, they inherit the default comparison behavior from :class:`object`. Types can customize their comparison behavior by implementing :dfn:`rich comparison methods` like :meth:`__lt__`, described in :ref:`customization`. The default behavior for equality comparison (``==`` and ``!=``) is based on the identity of the objects. Hence, equality comparison of instances with the same identity results in equality, and equality comparison of instances with different identities results in inequality. A motivation for this default behavior is the desire that all objects should be reflexive (i.e. ``x is y`` implies ``x == y``). A default order comparison (``<``, ``>``, ``<=``, and ``>=``) is not provided; an attempt raises :exc:`TypeError`. A motivation for this default behavior is the lack of a similar invariant as for equality. The behavior of the default equality comparison, that instances with different identities are always unequal, may be in contrast to what types will need that have a sensible definition of object value and value-based equality. Such types will need to customize their comparison behavior, and in fact, a number of built-in types have done that. The following list describes the comparison behavior of the most important built-in types. * Numbers of built-in numeric types (:ref:`typesnumeric`) and of the standard library types :class:`fractions.Fraction` and :class:`decimal.Decimal` can be compared within and across their types, with the restriction that complex numbers do not support order comparison. Within the limits of the types involved, they compare mathematically (algorithmically) correct without loss of precision. The not-a-number values :const:`float('NaN')` and :const:`Decimal('NaN')` are special. They are identical to themselves (``x is x`` is true) but are not equal to themselves (``x == x`` is false). Additionally, comparing any number to a not-a-number value will return ``False``. For example, both ``3 < float('NaN')`` and ``float('NaN') < 3`` will return ``False``. * Binary sequences (instances of :class:`bytes` or :class:`bytearray`) can be compared within and across their types. They compare lexicographically using the numeric values of their elements. * Strings (instances of :class:`str`) compare lexicographically using the numerical Unicode code points (the result of the built-in function :func:`ord`) of their characters. [#]_ Strings and binary sequences cannot be directly compared. * Sequences (instances of :class:`tuple`, :class:`list`, or :class:`range`) can be compared only within each of their types, with the restriction that ranges do not support order comparison. Equality comparison across these types results in inequality, and ordering comparison across these types raises :exc:`TypeError`. Sequences compare lexicographically using comparison of corresponding elements, whereby reflexivity of the elements is enforced. In enforcing reflexivity of elements, the comparison of collections assumes that for a collection element ``x``, ``x == x`` is always true. Based on that assumption, element identity is compared first, and element comparison is performed only for distinct elements. This approach yields the same result as a strict element comparison would, if the compared elements are reflexive. For non-reflexive elements, the result is different than for strict element comparison, and may be surprising: The non-reflexive not-a-number values for example result in the following comparison behavior when used in a list:: >>> nan = float('NaN') >>> nan is nan True >>> nan == nan False <-- the defined non-reflexive behavior of NaN >>> [nan] == [nan] True <-- list enforces reflexivity and tests identity first Lexicographical comparison between built-in collections works as follows: - For two collections to compare equal, they must be of the same type, have the same length, and each pair of corresponding elements must compare equal (for example, ``[1,2] == (1,2)`` is false because the type is not the same). - Collections that support order comparison are ordered the same as their first unequal elements (for example, ``[1,2,x] <= [1,2,y]`` has the same value as ``x <= y``). If a corresponding element does not exist, the shorter collection is ordered first (for example, ``[1,2] < [1,2,3]`` is true). * Mappings (instances of :class:`dict`) compare equal if and only if they have equal `(key, value)` pairs. Equality comparison of the keys and values enforces reflexivity. Order comparisons (``<``, ``>``, ``<=``, and ``>=``) raise :exc:`TypeError`. * Sets (instances of :class:`set` or :class:`frozenset`) can be compared within and across their types. They define order comparison operators to mean subset and superset tests. Those relations do not define total orderings (for example, the two sets ``{1,2}`` and ``{2,3}`` are not equal, nor subsets of one another, nor supersets of one another). Accordingly, sets are not appropriate arguments for functions which depend on total ordering (for example, :func:`min`, :func:`max`, and :func:`sorted` produce undefined results given a list of sets as inputs). Comparison of sets enforces reflexivity of its elements. * Most other built-in types have no comparison methods implemented, so they inherit the default comparison behavior. User-defined classes that customize their comparison behavior should follow some consistency rules, if possible: * Equality comparison should be reflexive. In other words, identical objects should compare equal: ``x is y`` implies ``x == y`` * Comparison should be symmetric. In other words, the following expressions should have the same result: ``x == y`` and ``y == x`` ``x != y`` and ``y != x`` ``x < y`` and ``y > x`` ``x <= y`` and ``y >= x`` * Comparison should be transitive. The following (non-exhaustive) examples illustrate that: ``x > y and y > z`` implies ``x > z`` ``x < y and y <= z`` implies ``x < z`` * Inverse comparison should result in the boolean negation. In other words, the following expressions should have the same result: ``x == y`` and ``not x != y`` ``x < y`` and ``not x >= y`` (for total ordering) ``x > y`` and ``not x <= y`` (for total ordering) The last two expressions apply to totally ordered collections (e.g. to sequences, but not to sets or mappings). See also the :func:`~functools.total_ordering` decorator. * The :func:`hash` result should be consistent with equality. Objects that are equal should either have the same hash value, or be marked as unhashable. Python does not enforce these consistency rules. In fact, the not-a-number values are an example for not following these rules. .. _in: .. _not in: .. _membership-test-details: Membership test operations -------------------------- The operators :keyword:`in` and :keyword:`not in` test for membership. ``x in s`` evaluates to ``True`` if *x* is a member of *s*, and ``False`` otherwise. ``x not in s`` returns the negation of ``x in s``. All built-in sequences and set types support this as well as dictionary, for which :keyword:`in` tests whether the dictionary has a given key. For container types such as list, tuple, set, frozenset, dict, or collections.deque, the expression ``x in y`` is equivalent to ``any(x is e or x == e for e in y)``. For the string and bytes types, ``x in y`` is ``True`` if and only if *x* is a substring of *y*. An equivalent test is ``y.find(x) != -1``. Empty strings are always considered to be a substring of any other string, so ``"" in "abc"`` will return ``True``. For user-defined classes which define the :meth:`__contains__` method, ``x in y`` returns ``True`` if ``y.__contains__(x)`` returns a true value, and ``False`` otherwise. For user-defined classes which do not define :meth:`__contains__` but do define :meth:`__iter__`, ``x in y`` is ``True`` if some value ``z`` with ``x == z`` is produced while iterating over ``y``. If an exception is raised during the iteration, it is as if :keyword:`in` raised that exception. Lastly, the old-style iteration protocol is tried: if a class defines :meth:`__getitem__`, ``x in y`` is ``True`` if and only if there is a non-negative integer index *i* such that ``x == y[i]``, and all lower integer indices do not raise :exc:`IndexError` exception. (If any other exception is raised, it is as if :keyword:`in` raised that exception). .. index:: operator: in operator: not in pair: membership; test object: sequence The operator :keyword:`not in` is defined to have the inverse true value of :keyword:`in`. .. index:: operator: is operator: is not pair: identity; test .. _is: .. _is not: Identity comparisons -------------------- The operators :keyword:`is` and :keyword:`is not` test for object identity: ``x is y`` is true if and only if *x* and *y* are the same object. Object identity is determined using the :meth:`id` function. ``x is not y`` yields the inverse truth value. [#]_ .. _booleans: .. _and: .. _or: .. _not: Boolean operations ================== .. index:: pair: Conditional; expression pair: Boolean; operation .. productionlist:: or_test: `and_test` | `or_test` "or" `and_test` and_test: `not_test` | `and_test` "and" `not_test` not_test: `comparison` | "not" `not_test` In the context of Boolean operations, and also when expressions are used by control flow statements, the following values are interpreted as false: ``False``, ``None``, numeric zero of all types, and empty strings and containers (including strings, tuples, lists, dictionaries, sets and frozensets). All other values are interpreted as true. User-defined objects can customize their truth value by providing a :meth:`__bool__` method. .. index:: operator: not The operator :keyword:`not` yields ``True`` if its argument is false, ``False`` otherwise. .. index:: operator: and The expression ``x and y`` first evaluates *x*; if *x* is false, its value is returned; otherwise, *y* is evaluated and the resulting value is returned. .. index:: operator: or The expression ``x or y`` first evaluates *x*; if *x* is true, its value is returned; otherwise, *y* is evaluated and the resulting value is returned. (Note that neither :keyword:`and` nor :keyword:`or` restrict the value and type they return to ``False`` and ``True``, but rather return the last evaluated argument. This is sometimes useful, e.g., if ``s`` is a string that should be replaced by a default value if it is empty, the expression ``s or 'foo'`` yields the desired value. Because :keyword:`not` has to create a new value, it returns a boolean value regardless of the type of its argument (for example, ``not 'foo'`` produces ``False`` rather than ``''``.) Conditional expressions ======================= .. index:: pair: conditional; expression pair: ternary; operator .. productionlist:: conditional_expression: `or_test` ["if" `or_test` "else" `expression`] expression: `conditional_expression` | `lambda_expr` expression_nocond: `or_test` | `lambda_expr_nocond` Conditional expressions (sometimes called a "ternary operator") have the lowest priority of all Python operations. The expression ``x if C else y`` first evaluates the condition, *C* rather than *x*. If *C* is true, *x* is evaluated and its value is returned; otherwise, *y* is evaluated and its value is returned. See :pep:`308` for more details about conditional expressions. .. _lambdas: .. _lambda: Lambdas ======= .. index:: pair: lambda; expression pair: lambda; form pair: anonymous; function .. productionlist:: lambda_expr: "lambda" [`parameter_list`]: `expression` lambda_expr_nocond: "lambda" [`parameter_list`]: `expression_nocond` Lambda expressions (sometimes called lambda forms) are used to create anonymous functions. The expression ``lambda parameters: expression`` yields a function object. The unnamed object behaves like a function object defined with: .. code-block:: none def (parameters): return expression See section :ref:`function` for the syntax of parameter lists. Note that functions created with lambda expressions cannot contain statements or annotations. .. _exprlists: Expression lists ================ .. index:: pair: expression; list .. productionlist:: expression_list: `expression` ("," `expression`)* [","] starred_list: `starred_item` ("," `starred_item`)* [","] starred_expression: `expression` | (`starred_item` ",")* [`starred_item`] starred_item: `expression` | "*" `or_expr` .. index:: object: tuple Except when part of a list or set display, an expression list containing at least one comma yields a tuple. The length of the tuple is the number of expressions in the list. The expressions are evaluated from left to right. .. index:: pair: iterable; unpacking single: *; in expression lists An asterisk ``*`` denotes :dfn:`iterable unpacking`. Its operand must be an :term:`iterable`. The iterable is expanded into a sequence of items, which are included in the new tuple, list, or set, at the site of the unpacking. .. versionadded:: 3.5 Iterable unpacking in expression lists, originally proposed by :pep:`448`. .. index:: pair: trailing; comma The trailing comma is required only to create a single tuple (a.k.a. a *singleton*); it is optional in all other cases. A single expression without a trailing comma doesn't create a tuple, but rather yields the value of that expression. (To create an empty tuple, use an empty pair of parentheses: ``()``.) .. _evalorder: Evaluation order ================ .. index:: pair: evaluation; order Python evaluates expressions from left to right. Notice that while evaluating an assignment, the right-hand side is evaluated before the left-hand side. In the following lines, expressions will be evaluated in the arithmetic order of their suffixes:: expr1, expr2, expr3, expr4 (expr1, expr2, expr3, expr4) {expr1: expr2, expr3: expr4} expr1 + expr2 * (expr3 - expr4) expr1(expr2, expr3, *expr4, **expr5) expr3, expr4 = expr1, expr2 .. _operator-summary: Operator precedence =================== .. index:: pair: operator; precedence The following table summarizes the operator precedence in Python, from lowest precedence (least binding) to highest precedence (most binding). Operators in the same box have the same precedence. Unless the syntax is explicitly given, operators are binary. Operators in the same box group left to right (except for exponentiation, which groups from right to left). Note that comparisons, membership tests, and identity tests, all have the same precedence and have a left-to-right chaining feature as described in the :ref:`comparisons` section. +-----------------------------------------------+-------------------------------------+ | Operator | Description | +===============================================+=====================================+ | :keyword:`lambda` | Lambda expression | +-----------------------------------------------+-------------------------------------+ | :keyword:`if` -- :keyword:`else` | Conditional expression | +-----------------------------------------------+-------------------------------------+ | :keyword:`or` | Boolean OR | +-----------------------------------------------+-------------------------------------+ | :keyword:`and` | Boolean AND | +-----------------------------------------------+-------------------------------------+ | :keyword:`not` ``x`` | Boolean NOT | +-----------------------------------------------+-------------------------------------+ | :keyword:`in`, :keyword:`not in`, | Comparisons, including membership | | :keyword:`is`, :keyword:`is not`, ``<``, | tests and identity tests | | ``<=``, ``>``, ``>=``, ``!=``, ``==`` | | +-----------------------------------------------+-------------------------------------+ | ``|`` | Bitwise OR | +-----------------------------------------------+-------------------------------------+ | ``^`` | Bitwise XOR | +-----------------------------------------------+-------------------------------------+ | ``&`` | Bitwise AND | +-----------------------------------------------+-------------------------------------+ | ``<<``, ``>>`` | Shifts | +-----------------------------------------------+-------------------------------------+ | ``+``, ``-`` | Addition and subtraction | +-----------------------------------------------+-------------------------------------+ | ``*``, ``@``, ``/``, ``//``, ``%`` | Multiplication, matrix | | | multiplication, division, floor | | | division, remainder [#]_ | +-----------------------------------------------+-------------------------------------+ | ``+x``, ``-x``, ``~x`` | Positive, negative, bitwise NOT | +-----------------------------------------------+-------------------------------------+ | ``**`` | Exponentiation [#]_ | +-----------------------------------------------+-------------------------------------+ | ``await`` ``x`` | Await expression | +-----------------------------------------------+-------------------------------------+ | ``x[index]``, ``x[index:index]``, | Subscription, slicing, | | ``x(arguments...)``, ``x.attribute`` | call, attribute reference | +-----------------------------------------------+-------------------------------------+ | ``(expressions...)``, | Binding or tuple display, | | ``[expressions...]``, | list display, | | ``{key: value...}``, | dictionary display, | | ``{expressions...}`` | set display | +-----------------------------------------------+-------------------------------------+ .. rubric:: Footnotes .. [#] While ``abs(x%y) < abs(y)`` is true mathematically, for floats it may not be true numerically due to roundoff. For example, and assuming a platform on which a Python float is an IEEE 754 double-precision number, in order that ``-1e-100 % 1e100`` have the same sign as ``1e100``, the computed result is ``-1e-100 + 1e100``, which is numerically exactly equal to ``1e100``. The function :func:`math.fmod` returns a result whose sign matches the sign of the first argument instead, and so returns ``-1e-100`` in this case. Which approach is more appropriate depends on the application. .. [#] If x is very close to an exact integer multiple of y, it's possible for ``x//y`` to be one larger than ``(x-x%y)//y`` due to rounding. In such cases, Python returns the latter result, in order to preserve that ``divmod(x,y)[0] * y + x % y`` be very close to ``x``. .. [#] The Unicode standard distinguishes between :dfn:`code points` (e.g. U+0041) and :dfn:`abstract characters` (e.g. "LATIN CAPITAL LETTER A"). While most abstract characters in Unicode are only represented using one code point, there is a number of abstract characters that can in addition be represented using a sequence of more than one code point. For example, the abstract character "LATIN CAPITAL LETTER C WITH CEDILLA" can be represented as a single :dfn:`precomposed character` at code position U+00C7, or as a sequence of a :dfn:`base character` at code position U+0043 (LATIN CAPITAL LETTER C), followed by a :dfn:`combining character` at code position U+0327 (COMBINING CEDILLA). The comparison operators on strings compare at the level of Unicode code points. This may be counter-intuitive to humans. For example, ``"\u00C7" == "\u0043\u0327"`` is ``False``, even though both strings represent the same abstract character "LATIN CAPITAL LETTER C WITH CEDILLA". To compare strings at the level of abstract characters (that is, in a way intuitive to humans), use :func:`unicodedata.normalize`. .. [#] Due to automatic garbage-collection, free lists, and the dynamic nature of descriptors, you may notice seemingly unusual behaviour in certain uses of the :keyword:`is` operator, like those involving comparisons between instance methods, or constants. Check their documentation for more info. .. [#] The ``%`` operator is also used for string formatting; the same precedence applies. .. [#] The power operator ``**`` binds less tightly than an arithmetic or bitwise unary operator on its right, that is, ``2**-1`` is ``0.5``. PK 3]ז _reference/simple_stmts.rst.txtnu[ .. _simple: ***************** Simple statements ***************** .. index:: pair: simple; statement A simple statement is comprised within a single logical line. Several simple statements may occur on a single line separated by semicolons. The syntax for simple statements is: .. productionlist:: simple_stmt: `expression_stmt` : | `assert_stmt` : | `assignment_stmt` : | `augmented_assignment_stmt` : | `annotated_assignment_stmt` : | `pass_stmt` : | `del_stmt` : | `return_stmt` : | `yield_stmt` : | `raise_stmt` : | `break_stmt` : | `continue_stmt` : | `import_stmt` : | `global_stmt` : | `nonlocal_stmt` .. _exprstmts: Expression statements ===================== .. index:: pair: expression; statement pair: expression; list .. index:: pair: expression; list Expression statements are used (mostly interactively) to compute and write a value, or (usually) to call a procedure (a function that returns no meaningful result; in Python, procedures return the value ``None``). Other uses of expression statements are allowed and occasionally useful. The syntax for an expression statement is: .. productionlist:: expression_stmt: `starred_expression` An expression statement evaluates the expression list (which may be a single expression). .. index:: builtin: repr object: None pair: string; conversion single: output pair: standard; output pair: writing; values pair: procedure; call In interactive mode, if the value is not ``None``, it is converted to a string using the built-in :func:`repr` function and the resulting string is written to standard output on a line by itself (except if the result is ``None``, so that procedure calls do not cause any output.) .. _assignment: Assignment statements ===================== .. index:: single: =; assignment statement pair: assignment; statement pair: binding; name pair: rebinding; name object: mutable pair: attribute; assignment Assignment statements are used to (re)bind names to values and to modify attributes or items of mutable objects: .. productionlist:: assignment_stmt: (`target_list` "=")+ (`starred_expression` | `yield_expression`) target_list: `target` ("," `target`)* [","] target: `identifier` : | "(" [`target_list`] ")" : | "[" [`target_list`] "]" : | `attributeref` : | `subscription` : | `slicing` : | "*" `target` (See section :ref:`primaries` for the syntax definitions for *attributeref*, *subscription*, and *slicing*.) An assignment statement evaluates the expression list (remember that this can be a single expression or a comma-separated list, the latter yielding a tuple) and assigns the single resulting object to each of the target lists, from left to right. .. index:: single: target pair: target; list Assignment is defined recursively depending on the form of the target (list). When a target is part of a mutable object (an attribute reference, subscription or slicing), the mutable object must ultimately perform the assignment and decide about its validity, and may raise an exception if the assignment is unacceptable. The rules observed by various types and the exceptions raised are given with the definition of the object types (see section :ref:`types`). .. index:: triple: target; list; assignment Assignment of an object to a target list, optionally enclosed in parentheses or square brackets, is recursively defined as follows. * If the target list is empty: The object must also be an empty iterable. * If the target list is a single target in parentheses: The object is assigned to that target. * If the target list is a comma-separated list of targets, or a single target in square brackets: The object must be an iterable with the same number of items as there are targets in the target list, and the items are assigned, from left to right, to the corresponding targets. * If the target list contains one target prefixed with an asterisk, called a "starred" target: The object must be an iterable with at least as many items as there are targets in the target list, minus one. The first items of the iterable are assigned, from left to right, to the targets before the starred target. The final items of the iterable are assigned to the targets after the starred target. A list of the remaining items in the iterable is then assigned to the starred target (the list can be empty). * Else: The object must be an iterable with the same number of items as there are targets in the target list, and the items are assigned, from left to right, to the corresponding targets. Assignment of an object to a single target is recursively defined as follows. * If the target is an identifier (name): * If the name does not occur in a :keyword:`global` or :keyword:`nonlocal` statement in the current code block: the name is bound to the object in the current local namespace. * Otherwise: the name is bound to the object in the global namespace or the outer namespace determined by :keyword:`nonlocal`, respectively. .. index:: single: destructor The name is rebound if it was already bound. This may cause the reference count for the object previously bound to the name to reach zero, causing the object to be deallocated and its destructor (if it has one) to be called. .. index:: pair: attribute; assignment * If the target is an attribute reference: The primary expression in the reference is evaluated. It should yield an object with assignable attributes; if this is not the case, :exc:`TypeError` is raised. That object is then asked to assign the assigned object to the given attribute; if it cannot perform the assignment, it raises an exception (usually but not necessarily :exc:`AttributeError`). .. _attr-target-note: Note: If the object is a class instance and the attribute reference occurs on both sides of the assignment operator, the RHS expression, ``a.x`` can access either an instance attribute or (if no instance attribute exists) a class attribute. The LHS target ``a.x`` is always set as an instance attribute, creating it if necessary. Thus, the two occurrences of ``a.x`` do not necessarily refer to the same attribute: if the RHS expression refers to a class attribute, the LHS creates a new instance attribute as the target of the assignment:: class Cls: x = 3 # class variable inst = Cls() inst.x = inst.x + 1 # writes inst.x as 4 leaving Cls.x as 3 This description does not necessarily apply to descriptor attributes, such as properties created with :func:`property`. .. index:: pair: subscription; assignment object: mutable * If the target is a subscription: The primary expression in the reference is evaluated. It should yield either a mutable sequence object (such as a list) or a mapping object (such as a dictionary). Next, the subscript expression is evaluated. .. index:: object: sequence object: list If the primary is a mutable sequence object (such as a list), the subscript must yield an integer. If it is negative, the sequence's length is added to it. The resulting value must be a nonnegative integer less than the sequence's length, and the sequence is asked to assign the assigned object to its item with that index. If the index is out of range, :exc:`IndexError` is raised (assignment to a subscripted sequence cannot add new items to a list). .. index:: object: mapping object: dictionary If the primary is a mapping object (such as a dictionary), the subscript must have a type compatible with the mapping's key type, and the mapping is then asked to create a key/datum pair which maps the subscript to the assigned object. This can either replace an existing key/value pair with the same key value, or insert a new key/value pair (if no key with the same value existed). For user-defined objects, the :meth:`__setitem__` method is called with appropriate arguments. .. index:: pair: slicing; assignment * If the target is a slicing: The primary expression in the reference is evaluated. It should yield a mutable sequence object (such as a list). The assigned object should be a sequence object of the same type. Next, the lower and upper bound expressions are evaluated, insofar they are present; defaults are zero and the sequence's length. The bounds should evaluate to integers. If either bound is negative, the sequence's length is added to it. The resulting bounds are clipped to lie between zero and the sequence's length, inclusive. Finally, the sequence object is asked to replace the slice with the items of the assigned sequence. The length of the slice may be different from the length of the assigned sequence, thus changing the length of the target sequence, if the target sequence allows it. .. impl-detail:: In the current implementation, the syntax for targets is taken to be the same as for expressions, and invalid syntax is rejected during the code generation phase, causing less detailed error messages. Although the definition of assignment implies that overlaps between the left-hand side and the right-hand side are 'simultaneous' (for example ``a, b = b, a`` swaps two variables), overlaps *within* the collection of assigned-to variables occur left-to-right, sometimes resulting in confusion. For instance, the following program prints ``[0, 2]``:: x = [0, 1] i = 0 i, x[i] = 1, 2 # i is updated, then x[i] is updated print(x) .. seealso:: :pep:`3132` - Extended Iterable Unpacking The specification for the ``*target`` feature. .. _augassign: Augmented assignment statements ------------------------------- .. index:: pair: augmented; assignment single: statement; assignment, augmented single: +=; augmented assignment single: -=; augmented assignment single: *=; augmented assignment single: /=; augmented assignment single: %=; augmented assignment single: &=; augmented assignment single: ^=; augmented assignment single: |=; augmented assignment single: **=; augmented assignment single: //=; augmented assignment single: >>=; augmented assignment single: <<=; augmented assignment Augmented assignment is the combination, in a single statement, of a binary operation and an assignment statement: .. productionlist:: augmented_assignment_stmt: `augtarget` `augop` (`expression_list` | `yield_expression`) augtarget: `identifier` | `attributeref` | `subscription` | `slicing` augop: "+=" | "-=" | "*=" | "@=" | "/=" | "//=" | "%=" | "**=" : | ">>=" | "<<=" | "&=" | "^=" | "|=" (See section :ref:`primaries` for the syntax definitions of the last three symbols.) An augmented assignment evaluates the target (which, unlike normal assignment statements, cannot be an unpacking) and the expression list, performs the binary operation specific to the type of assignment on the two operands, and assigns the result to the original target. The target is only evaluated once. An augmented assignment expression like ``x += 1`` can be rewritten as ``x = x + 1`` to achieve a similar, but not exactly equal effect. In the augmented version, ``x`` is only evaluated once. Also, when possible, the actual operation is performed *in-place*, meaning that rather than creating a new object and assigning that to the target, the old object is modified instead. Unlike normal assignments, augmented assignments evaluate the left-hand side *before* evaluating the right-hand side. For example, ``a[i] += f(x)`` first looks-up ``a[i]``, then it evaluates ``f(x)`` and performs the addition, and lastly, it writes the result back to ``a[i]``. With the exception of assigning to tuples and multiple targets in a single statement, the assignment done by augmented assignment statements is handled the same way as normal assignments. Similarly, with the exception of the possible *in-place* behavior, the binary operation performed by augmented assignment is the same as the normal binary operations. For targets which are attribute references, the same :ref:`caveat about class and instance attributes ` applies as for regular assignments. .. _annassign: Annotated assignment statements ------------------------------- .. index:: pair: annotated; assignment single: statement; assignment, annotated Annotation assignment is the combination, in a single statement, of a variable or attribute annotation and an optional assignment statement: .. productionlist:: annotated_assignment_stmt: `augtarget` ":" `expression` ["=" `expression`] The difference from normal :ref:`assignment` is that only single target and only single right hand side value is allowed. For simple names as assignment targets, if in class or module scope, the annotations are evaluated and stored in a special class or module attribute :attr:`__annotations__` that is a dictionary mapping from variable names (mangled if private) to evaluated annotations. This attribute is writable and is automatically created at the start of class or module body execution, if annotations are found statically. For expressions as assignment targets, the annotations are evaluated if in class or module scope, but not stored. If a name is annotated in a function scope, then this name is local for that scope. Annotations are never evaluated and stored in function scopes. If the right hand side is present, an annotated assignment performs the actual assignment before evaluating annotations (where applicable). If the right hand side is not present for an expression target, then the interpreter evaluates the target except for the last :meth:`__setitem__` or :meth:`__setattr__` call. .. seealso:: :pep:`526` - Variable and attribute annotation syntax :pep:`484` - Type hints .. _assert: The :keyword:`assert` statement =============================== .. index:: statement: assert pair: debugging; assertions Assert statements are a convenient way to insert debugging assertions into a program: .. productionlist:: assert_stmt: "assert" `expression` ["," `expression`] The simple form, ``assert expression``, is equivalent to :: if __debug__: if not expression: raise AssertionError The extended form, ``assert expression1, expression2``, is equivalent to :: if __debug__: if not expression1: raise AssertionError(expression2) .. index:: single: __debug__ exception: AssertionError These equivalences assume that :const:`__debug__` and :exc:`AssertionError` refer to the built-in variables with those names. In the current implementation, the built-in variable :const:`__debug__` is ``True`` under normal circumstances, ``False`` when optimization is requested (command line option -O). The current code generator emits no code for an assert statement when optimization is requested at compile time. Note that it is unnecessary to include the source code for the expression that failed in the error message; it will be displayed as part of the stack trace. Assignments to :const:`__debug__` are illegal. The value for the built-in variable is determined when the interpreter starts. .. _pass: The :keyword:`pass` statement ============================= .. index:: statement: pass pair: null; operation pair: null; operation .. productionlist:: pass_stmt: "pass" :keyword:`pass` is a null operation --- when it is executed, nothing happens. It is useful as a placeholder when a statement is required syntactically, but no code needs to be executed, for example:: def f(arg): pass # a function that does nothing (yet) class C: pass # a class with no methods (yet) .. _del: The :keyword:`del` statement ============================ .. index:: statement: del pair: deletion; target triple: deletion; target; list .. productionlist:: del_stmt: "del" `target_list` Deletion is recursively defined very similar to the way assignment is defined. Rather than spelling it out in full details, here are some hints. Deletion of a target list recursively deletes each target, from left to right. .. index:: statement: global pair: unbinding; name Deletion of a name removes the binding of that name from the local or global namespace, depending on whether the name occurs in a :keyword:`global` statement in the same code block. If the name is unbound, a :exc:`NameError` exception will be raised. .. index:: pair: attribute; deletion Deletion of attribute references, subscriptions and slicings is passed to the primary object involved; deletion of a slicing is in general equivalent to assignment of an empty slice of the right type (but even this is determined by the sliced object). .. versionchanged:: 3.2 Previously it was illegal to delete a name from the local namespace if it occurs as a free variable in a nested block. .. _return: The :keyword:`return` statement =============================== .. index:: statement: return pair: function; definition pair: class; definition .. productionlist:: return_stmt: "return" [`expression_list`] :keyword:`return` may only occur syntactically nested in a function definition, not within a nested class definition. If an expression list is present, it is evaluated, else ``None`` is substituted. :keyword:`return` leaves the current function call with the expression list (or ``None``) as return value. .. index:: keyword: finally When :keyword:`return` passes control out of a :keyword:`try` statement with a :keyword:`finally` clause, that :keyword:`finally` clause is executed before really leaving the function. In a generator function, the :keyword:`return` statement indicates that the generator is done and will cause :exc:`StopIteration` to be raised. The returned value (if any) is used as an argument to construct :exc:`StopIteration` and becomes the :attr:`StopIteration.value` attribute. In an asynchronous generator function, an empty :keyword:`return` statement indicates that the asynchronous generator is done and will cause :exc:`StopAsyncIteration` to be raised. A non-empty :keyword:`return` statement is a syntax error in an asynchronous generator function. .. _yield: The :keyword:`yield` statement ============================== .. index:: statement: yield single: generator; function single: generator; iterator single: function; generator exception: StopIteration .. productionlist:: yield_stmt: `yield_expression` A :keyword:`yield` statement is semantically equivalent to a :ref:`yield expression `. The yield statement can be used to omit the parentheses that would otherwise be required in the equivalent yield expression statement. For example, the yield statements :: yield yield from are equivalent to the yield expression statements :: (yield ) (yield from ) Yield expressions and statements are only used when defining a :term:`generator` function, and are only used in the body of the generator function. Using yield in a function definition is sufficient to cause that definition to create a generator function instead of a normal function. For full details of :keyword:`yield` semantics, refer to the :ref:`yieldexpr` section. .. _raise: The :keyword:`raise` statement ============================== .. index:: statement: raise single: exception pair: raising; exception single: __traceback__ (exception attribute) .. productionlist:: raise_stmt: "raise" [`expression` ["from" `expression`]] If no expressions are present, :keyword:`raise` re-raises the last exception that was active in the current scope. If no exception is active in the current scope, a :exc:`RuntimeError` exception is raised indicating that this is an error. Otherwise, :keyword:`raise` evaluates the first expression as the exception object. It must be either a subclass or an instance of :class:`BaseException`. If it is a class, the exception instance will be obtained when needed by instantiating the class with no arguments. The :dfn:`type` of the exception is the exception instance's class, the :dfn:`value` is the instance itself. .. index:: object: traceback A traceback object is normally created automatically when an exception is raised and attached to it as the :attr:`__traceback__` attribute, which is writable. You can create an exception and set your own traceback in one step using the :meth:`with_traceback` exception method (which returns the same exception instance, with its traceback set to its argument), like so:: raise Exception("foo occurred").with_traceback(tracebackobj) .. index:: pair: exception; chaining __cause__ (exception attribute) __context__ (exception attribute) The ``from`` clause is used for exception chaining: if given, the second *expression* must be another exception class or instance, which will then be attached to the raised exception as the :attr:`__cause__` attribute (which is writable). If the raised exception is not handled, both exceptions will be printed:: >>> try: ... print(1 / 0) ... except Exception as exc: ... raise RuntimeError("Something bad happened") from exc ... Traceback (most recent call last): File "", line 2, in ZeroDivisionError: division by zero The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 4, in RuntimeError: Something bad happened A similar mechanism works implicitly if an exception is raised inside an exception handler or a :keyword:`finally` clause: the previous exception is then attached as the new exception's :attr:`__context__` attribute:: >>> try: ... print(1 / 0) ... except: ... raise RuntimeError("Something bad happened") ... Traceback (most recent call last): File "", line 2, in ZeroDivisionError: division by zero During handling of the above exception, another exception occurred: Traceback (most recent call last): File "", line 4, in RuntimeError: Something bad happened Exception chaining can be explicitly suppressed by specifying :const:`None` in the ``from`` clause:: >>> try: ... print(1 / 0) ... except: ... raise RuntimeError("Something bad happened") from None ... Traceback (most recent call last): File "", line 4, in RuntimeError: Something bad happened Additional information on exceptions can be found in section :ref:`exceptions`, and information about handling exceptions is in section :ref:`try`. .. versionchanged:: 3.3 :const:`None` is now permitted as ``Y`` in ``raise X from Y``. .. versionadded:: 3.3 The ``__suppress_context__`` attribute to suppress automatic display of the exception context. .. _break: The :keyword:`break` statement ============================== .. index:: statement: break statement: for statement: while pair: loop; statement .. productionlist:: break_stmt: "break" :keyword:`break` may only occur syntactically nested in a :keyword:`for` or :keyword:`while` loop, but not nested in a function or class definition within that loop. .. index:: keyword: else pair: loop control; target It terminates the nearest enclosing loop, skipping the optional :keyword:`else` clause if the loop has one. If a :keyword:`for` loop is terminated by :keyword:`break`, the loop control target keeps its current value. .. index:: keyword: finally When :keyword:`break` passes control out of a :keyword:`try` statement with a :keyword:`finally` clause, that :keyword:`finally` clause is executed before really leaving the loop. .. _continue: The :keyword:`continue` statement ================================= .. index:: statement: continue statement: for statement: while pair: loop; statement keyword: finally .. productionlist:: continue_stmt: "continue" :keyword:`continue` may only occur syntactically nested in a :keyword:`for` or :keyword:`while` loop, but not nested in a function or class definition or :keyword:`finally` clause within that loop. It continues with the next cycle of the nearest enclosing loop. When :keyword:`continue` passes control out of a :keyword:`try` statement with a :keyword:`finally` clause, that :keyword:`finally` clause is executed before really starting the next loop cycle. .. _import: .. _from: The :keyword:`import` statement =============================== .. index:: statement: import single: module; importing pair: name; binding keyword: from .. productionlist:: import_stmt: "import" `module` ["as" `identifier`] ("," `module` ["as" `identifier`])* : | "from" `relative_module` "import" `identifier` ["as" `identifier`] : ("," `identifier` ["as" `identifier`])* : | "from" `relative_module` "import" "(" `identifier` ["as" `identifier`] : ("," `identifier` ["as" `identifier`])* [","] ")" : | "from" `module` "import" "*" module: (`identifier` ".")* `identifier` relative_module: "."* `module` | "."+ The basic import statement (no :keyword:`from` clause) is executed in two steps: #. find a module, loading and initializing it if necessary #. define a name or names in the local namespace for the scope where the :keyword:`import` statement occurs. When the statement contains multiple clauses (separated by commas) the two steps are carried out separately for each clause, just as though the clauses had been separated out into individual import statements. The details of the first step, finding and loading modules are described in greater detail in the section on the :ref:`import system `, which also describes the various types of packages and modules that can be imported, as well as all the hooks that can be used to customize the import system. Note that failures in this step may indicate either that the module could not be located, *or* that an error occurred while initializing the module, which includes execution of the module's code. If the requested module is retrieved successfully, it will be made available in the local namespace in one of three ways: .. index:: single: as; import statement * If the module name is followed by :keyword:`as`, then the name following :keyword:`as` is bound directly to the imported module. * If no other name is specified, and the module being imported is a top level module, the module's name is bound in the local namespace as a reference to the imported module * If the module being imported is *not* a top level module, then the name of the top level package that contains the module is bound in the local namespace as a reference to the top level package. The imported module must be accessed using its full qualified name rather than directly .. index:: pair: name; binding keyword: from exception: ImportError The :keyword:`from` form uses a slightly more complex process: #. find the module specified in the :keyword:`from` clause, loading and initializing it if necessary; #. for each of the identifiers specified in the :keyword:`import` clauses: #. check if the imported module has an attribute by that name #. if not, attempt to import a submodule with that name and then check the imported module again for that attribute #. if the attribute is not found, :exc:`ImportError` is raised. #. otherwise, a reference to that value is stored in the local namespace, using the name in the :keyword:`as` clause if it is present, otherwise using the attribute name Examples:: import foo # foo imported and bound locally import foo.bar.baz # foo.bar.baz imported, foo bound locally import foo.bar.baz as fbb # foo.bar.baz imported and bound as fbb from foo.bar import baz # foo.bar.baz imported and bound as baz from foo import attr # foo imported and foo.attr bound as attr If the list of identifiers is replaced by a star (``'*'``), all public names defined in the module are bound in the local namespace for the scope where the :keyword:`import` statement occurs. .. index:: single: __all__ (optional module attribute) The *public names* defined by a module are determined by checking the module's namespace for a variable named ``__all__``; if defined, it must be a sequence of strings which are names defined or imported by that module. The names given in ``__all__`` are all considered public and are required to exist. If ``__all__`` is not defined, the set of public names includes all names found in the module's namespace which do not begin with an underscore character (``'_'``). ``__all__`` should contain the entire public API. It is intended to avoid accidentally exporting items that are not part of the API (such as library modules which were imported and used within the module). The wild card form of import --- ``from module import *`` --- is only allowed at the module level. Attempting to use it in class or function definitions will raise a :exc:`SyntaxError`. .. index:: single: relative; import When specifying what module to import you do not have to specify the absolute name of the module. When a module or package is contained within another package it is possible to make a relative import within the same top package without having to mention the package name. By using leading dots in the specified module or package after :keyword:`from` you can specify how high to traverse up the current package hierarchy without specifying exact names. One leading dot means the current package where the module making the import exists. Two dots means up one package level. Three dots is up two levels, etc. So if you execute ``from . import mod`` from a module in the ``pkg`` package then you will end up importing ``pkg.mod``. If you execute ``from ..subpkg2 import mod`` from within ``pkg.subpkg1`` you will import ``pkg.subpkg2.mod``. The specification for relative imports is contained within :pep:`328`. :func:`importlib.import_module` is provided to support applications that determine dynamically the modules to be loaded. .. _future: Future statements ----------------- .. index:: pair: future; statement A :dfn:`future statement` is a directive to the compiler that a particular module should be compiled using syntax or semantics that will be available in a specified future release of Python where the feature becomes standard. The future statement is intended to ease migration to future versions of Python that introduce incompatible changes to the language. It allows use of the new features on a per-module basis before the release in which the feature becomes standard. .. productionlist:: * future_stmt: "from" "__future__" "import" `feature` ["as" `identifier`] : ("," `feature` ["as" `identifier`])* : | "from" "__future__" "import" "(" `feature` ["as" `identifier`] : ("," `feature` ["as" `identifier`])* [","] ")" feature: `identifier` A future statement must appear near the top of the module. The only lines that can appear before a future statement are: * the module docstring (if any), * comments, * blank lines, and * other future statements. .. XXX change this if future is cleaned out The features recognized by Python 3.0 are ``absolute_import``, ``division``, ``generators``, ``unicode_literals``, ``print_function``, ``nested_scopes`` and ``with_statement``. They are all redundant because they are always enabled, and only kept for backwards compatibility. A future statement is recognized and treated specially at compile time: Changes to the semantics of core constructs are often implemented by generating different code. It may even be the case that a new feature introduces new incompatible syntax (such as a new reserved word), in which case the compiler may need to parse the module differently. Such decisions cannot be pushed off until runtime. For any given release, the compiler knows which feature names have been defined, and raises a compile-time error if a future statement contains a feature not known to it. The direct runtime semantics are the same as for any import statement: there is a standard module :mod:`__future__`, described later, and it will be imported in the usual way at the time the future statement is executed. The interesting runtime semantics depend on the specific feature enabled by the future statement. Note that there is nothing special about the statement:: import __future__ [as name] That is not a future statement; it's an ordinary import statement with no special semantics or syntax restrictions. Code compiled by calls to the built-in functions :func:`exec` and :func:`compile` that occur in a module :mod:`M` containing a future statement will, by default, use the new syntax or semantics associated with the future statement. This can be controlled by optional arguments to :func:`compile` --- see the documentation of that function for details. A future statement typed at an interactive interpreter prompt will take effect for the rest of the interpreter session. If an interpreter is started with the :option:`-i` option, is passed a script name to execute, and the script includes a future statement, it will be in effect in the interactive session started after the script is executed. .. seealso:: :pep:`236` - Back to the __future__ The original proposal for the __future__ mechanism. .. _global: The :keyword:`global` statement =============================== .. index:: statement: global triple: global; name; binding .. productionlist:: global_stmt: "global" `identifier` ("," `identifier`)* The :keyword:`global` statement is a declaration which holds for the entire current code block. It means that the listed identifiers are to be interpreted as globals. It would be impossible to assign to a global variable without :keyword:`global`, although free variables may refer to globals without being declared global. Names listed in a :keyword:`global` statement must not be used in the same code block textually preceding that :keyword:`global` statement. Names listed in a :keyword:`global` statement must not be defined as formal parameters or in a :keyword:`for` loop control target, :keyword:`class` definition, function definition, :keyword:`import` statement, or variable annotation. .. impl-detail:: The current implementation does not enforce some of these restrictions, but programs should not abuse this freedom, as future implementations may enforce them or silently change the meaning of the program. .. index:: builtin: exec builtin: eval builtin: compile **Programmer's note:** :keyword:`global` is a directive to the parser. It applies only to code parsed at the same time as the :keyword:`global` statement. In particular, a :keyword:`global` statement contained in a string or code object supplied to the built-in :func:`exec` function does not affect the code block *containing* the function call, and code contained in such a string is unaffected by :keyword:`global` statements in the code containing the function call. The same applies to the :func:`eval` and :func:`compile` functions. .. _nonlocal: The :keyword:`nonlocal` statement ================================= .. index:: statement: nonlocal .. productionlist:: nonlocal_stmt: "nonlocal" `identifier` ("," `identifier`)* .. XXX add when implemented : ["=" (`target_list` "=")+ starred_expression] : | "nonlocal" identifier augop expression_list The :keyword:`nonlocal` statement causes the listed identifiers to refer to previously bound variables in the nearest enclosing scope excluding globals. This is important because the default behavior for binding is to search the local namespace first. The statement allows encapsulated code to rebind variables outside of the local scope besides the global (module) scope. .. XXX not implemented The :keyword:`nonlocal` statement may prepend an assignment or augmented assignment, but not an expression. Names listed in a :keyword:`nonlocal` statement, unlike those listed in a :keyword:`global` statement, must refer to pre-existing bindings in an enclosing scope (the scope in which a new binding should be created cannot be determined unambiguously). Names listed in a :keyword:`nonlocal` statement must not collide with pre-existing bindings in the local scope. .. seealso:: :pep:`3104` - Access to Names in Outer Scopes The specification for the :keyword:`nonlocal` statement. PK 3]@* %reference/toplevel_components.rst.txtnu[ .. _top-level: ******************** Top-level components ******************** .. index:: single: interpreter The Python interpreter can get its input from a number of sources: from a script passed to it as standard input or as program argument, typed in interactively, from a module source file, etc. This chapter gives the syntax used in these cases. .. _programs: Complete Python programs ======================== .. index:: single: program .. index:: module: sys module: __main__ module: builtins While a language specification need not prescribe how the language interpreter is invoked, it is useful to have a notion of a complete Python program. A complete Python program is executed in a minimally initialized environment: all built-in and standard modules are available, but none have been initialized, except for :mod:`sys` (various system services), :mod:`builtins` (built-in functions, exceptions and ``None``) and :mod:`__main__`. The latter is used to provide the local and global namespace for execution of the complete program. The syntax for a complete Python program is that for file input, described in the next section. .. index:: single: interactive mode module: __main__ The interpreter may also be invoked in interactive mode; in this case, it does not read and execute a complete program but reads and executes one statement (possibly compound) at a time. The initial environment is identical to that of a complete program; each statement is executed in the namespace of :mod:`__main__`. .. index:: single: UNIX single: Windows single: command line single: standard input A complete program can be passed to the interpreter in three forms: with the :option:`-c` *string* command line option, as a file passed as the first command line argument, or as standard input. If the file or standard input is a tty device, the interpreter enters interactive mode; otherwise, it executes the file as a complete program. .. _file-input: File input ========== All input read from non-interactive files has the same form: .. productionlist:: file_input: (NEWLINE | `statement`)* This syntax is used in the following situations: * when parsing a complete Python program (from a file or from a string); * when parsing a module; * when parsing a string passed to the :func:`exec` function; .. _interactive: Interactive input ================= Input in interactive mode is parsed using the following grammar: .. productionlist:: interactive_input: [`stmt_list`] NEWLINE | `compound_stmt` NEWLINE Note that a (top-level) compound statement must be followed by a blank line in interactive mode; this is needed to help the parser detect the end of the input. .. _expression-input: Expression input ================ .. index:: single: input .. index:: builtin: eval :func:`eval` is used for expression input. It ignores leading whitespace. The string argument to :func:`eval` must have the following form: .. productionlist:: eval_input: `expression_list` NEWLINE* PK 3]6 reference/grammar.rst.txtnu[Full Grammar specification ========================== This is the full Python grammar, as it is read by the parser generator and used to parse Python source files: .. literalinclude:: ../../Grammar/Grammar PK 3]zreference/index.rst.txtnu[.. _reference-index: ################################# The Python Language Reference ################################# This reference manual describes the syntax and "core semantics" of the language. It is terse, but attempts to be exact and complete. The semantics of non-essential built-in object types and of the built-in functions and modules are described in :ref:`library-index`. For an informal introduction to the language, see :ref:`tutorial-index`. For C or C++ programmers, two additional manuals exist: :ref:`extending-index` describes the high-level picture of how to write a Python extension module, and the :ref:`c-api-index` describes the interfaces available to C/C++ programmers in detail. .. toctree:: :maxdepth: 2 :numbered: introduction.rst lexical_analysis.rst datamodel.rst executionmodel.rst import.rst expressions.rst simple_stmts.rst compound_stmts.rst toplevel_components.rst grammar.rst PK 3] reference/import.rst.txtnu[ .. _importsystem: ***************** The import system ***************** .. index:: single: import machinery Python code in one :term:`module` gains access to the code in another module by the process of :term:`importing` it. The :keyword:`import` statement is the most common way of invoking the import machinery, but it is not the only way. Functions such as :func:`importlib.import_module` and built-in :func:`__import__` can also be used to invoke the import machinery. The :keyword:`import` statement combines two operations; it searches for the named module, then it binds the results of that search to a name in the local scope. The search operation of the :keyword:`import` statement is defined as a call to the :func:`__import__` function, with the appropriate arguments. The return value of :func:`__import__` is used to perform the name binding operation of the :keyword:`import` statement. See the :keyword:`import` statement for the exact details of that name binding operation. A direct call to :func:`__import__` performs only the module search and, if found, the module creation operation. While certain side-effects may occur, such as the importing of parent packages, and the updating of various caches (including :data:`sys.modules`), only the :keyword:`import` statement performs a name binding operation. When calling :func:`__import__` as part of an import statement, the standard builtin :func:`__import__` is called. Other mechanisms for invoking the import system (such as :func:`importlib.import_module`) may choose to subvert :func:`__import__` and use its own solution to implement import semantics. When a module is first imported, Python searches for the module and if found, it creates a module object [#fnmo]_, initializing it. If the named module cannot be found, a :exc:`ModuleNotFoundError` is raised. Python implements various strategies to search for the named module when the import machinery is invoked. These strategies can be modified and extended by using various hooks described in the sections below. .. versionchanged:: 3.3 The import system has been updated to fully implement the second phase of :pep:`302`. There is no longer any implicit import machinery - the full import system is exposed through :data:`sys.meta_path`. In addition, native namespace package support has been implemented (see :pep:`420`). :mod:`importlib` ================ The :mod:`importlib` module provides a rich API for interacting with the import system. For example :func:`importlib.import_module` provides a recommended, simpler API than built-in :func:`__import__` for invoking the import machinery. Refer to the :mod:`importlib` library documentation for additional detail. Packages ======== .. index:: single: package Python has only one type of module object, and all modules are of this type, regardless of whether the module is implemented in Python, C, or something else. To help organize modules and provide a naming hierarchy, Python has a concept of :term:`packages `. You can think of packages as the directories on a file system and modules as files within directories, but don't take this analogy too literally since packages and modules need not originate from the file system. For the purposes of this documentation, we'll use this convenient analogy of directories and files. Like file system directories, packages are organized hierarchically, and packages may themselves contain subpackages, as well as regular modules. It's important to keep in mind that all packages are modules, but not all modules are packages. Or put another way, packages are just a special kind of module. Specifically, any module that contains a ``__path__`` attribute is considered a package. All modules have a name. Subpackage names are separated from their parent package name by dots, akin to Python's standard attribute access syntax. Thus you might have a module called :mod:`sys` and a package called :mod:`email`, which in turn has a subpackage called :mod:`email.mime` and a module within that subpackage called :mod:`email.mime.text`. Regular packages ---------------- .. index:: pair: package; regular Python defines two types of packages, :term:`regular packages ` and :term:`namespace packages `. Regular packages are traditional packages as they existed in Python 3.2 and earlier. A regular package is typically implemented as a directory containing an ``__init__.py`` file. When a regular package is imported, this ``__init__.py`` file is implicitly executed, and the objects it defines are bound to names in the package's namespace. The ``__init__.py`` file can contain the same Python code that any other module can contain, and Python will add some additional attributes to the module when it is imported. For example, the following file system layout defines a top level ``parent`` package with three subpackages:: parent/ __init__.py one/ __init__.py two/ __init__.py three/ __init__.py Importing ``parent.one`` will implicitly execute ``parent/__init__.py`` and ``parent/one/__init__.py``. Subsequent imports of ``parent.two`` or ``parent.three`` will execute ``parent/two/__init__.py`` and ``parent/three/__init__.py`` respectively. Namespace packages ------------------ .. index:: pair:: package; namespace pair:: package; portion A namespace package is a composite of various :term:`portions `, where each portion contributes a subpackage to the parent package. Portions may reside in different locations on the file system. Portions may also be found in zip files, on the network, or anywhere else that Python searches during import. Namespace packages may or may not correspond directly to objects on the file system; they may be virtual modules that have no concrete representation. Namespace packages do not use an ordinary list for their ``__path__`` attribute. They instead use a custom iterable type which will automatically perform a new search for package portions on the next import attempt within that package if the path of their parent package (or :data:`sys.path` for a top level package) changes. With namespace packages, there is no ``parent/__init__.py`` file. In fact, there may be multiple ``parent`` directories found during import search, where each one is provided by a different portion. Thus ``parent/one`` may not be physically located next to ``parent/two``. In this case, Python will create a namespace package for the top-level ``parent`` package whenever it or one of its subpackages is imported. See also :pep:`420` for the namespace package specification. Searching ========= To begin the search, Python needs the :term:`fully qualified ` name of the module (or package, but for the purposes of this discussion, the difference is immaterial) being imported. This name may come from various arguments to the :keyword:`import` statement, or from the parameters to the :func:`importlib.import_module` or :func:`__import__` functions. This name will be used in various phases of the import search, and it may be the dotted path to a submodule, e.g. ``foo.bar.baz``. In this case, Python first tries to import ``foo``, then ``foo.bar``, and finally ``foo.bar.baz``. If any of the intermediate imports fail, a :exc:`ModuleNotFoundError` is raised. The module cache ---------------- .. index:: single: sys.modules The first place checked during import search is :data:`sys.modules`. This mapping serves as a cache of all modules that have been previously imported, including the intermediate paths. So if ``foo.bar.baz`` was previously imported, :data:`sys.modules` will contain entries for ``foo``, ``foo.bar``, and ``foo.bar.baz``. Each key will have as its value the corresponding module object. During import, the module name is looked up in :data:`sys.modules` and if present, the associated value is the module satisfying the import, and the process completes. However, if the value is ``None``, then a :exc:`ModuleNotFoundError` is raised. If the module name is missing, Python will continue searching for the module. :data:`sys.modules` is writable. Deleting a key may not destroy the associated module (as other modules may hold references to it), but it will invalidate the cache entry for the named module, causing Python to search anew for the named module upon its next import. The key can also be assigned to ``None``, forcing the next import of the module to result in a :exc:`ModuleNotFoundError`. Beware though, as if you keep a reference to the module object, invalidate its cache entry in :data:`sys.modules`, and then re-import the named module, the two module objects will *not* be the same. By contrast, :func:`importlib.reload` will reuse the *same* module object, and simply reinitialise the module contents by rerunning the module's code. Finders and loaders ------------------- .. index:: single: finder single: loader single: module spec If the named module is not found in :data:`sys.modules`, then Python's import protocol is invoked to find and load the module. This protocol consists of two conceptual objects, :term:`finders ` and :term:`loaders `. A finder's job is to determine whether it can find the named module using whatever strategy it knows about. Objects that implement both of these interfaces are referred to as :term:`importers ` - they return themselves when they find that they can load the requested module. Python includes a number of default finders and importers. The first one knows how to locate built-in modules, and the second knows how to locate frozen modules. A third default finder searches an :term:`import path` for modules. The :term:`import path` is a list of locations that may name file system paths or zip files. It can also be extended to search for any locatable resource, such as those identified by URLs. The import machinery is extensible, so new finders can be added to extend the range and scope of module searching. Finders do not actually load modules. If they can find the named module, they return a :dfn:`module spec`, an encapsulation of the module's import-related information, which the import machinery then uses when loading the module. The following sections describe the protocol for finders and loaders in more detail, including how you can create and register new ones to extend the import machinery. .. versionchanged:: 3.4 In previous versions of Python, finders returned :term:`loaders ` directly, whereas now they return module specs which *contain* loaders. Loaders are still used during import but have fewer responsibilities. Import hooks ------------ .. index:: single: import hooks single: meta hooks single: path hooks pair: hooks; import pair: hooks; meta pair: hooks; path The import machinery is designed to be extensible; the primary mechanism for this are the *import hooks*. There are two types of import hooks: *meta hooks* and *import path hooks*. Meta hooks are called at the start of import processing, before any other import processing has occurred, other than :data:`sys.modules` cache look up. This allows meta hooks to override :data:`sys.path` processing, frozen modules, or even built-in modules. Meta hooks are registered by adding new finder objects to :data:`sys.meta_path`, as described below. Import path hooks are called as part of :data:`sys.path` (or ``package.__path__``) processing, at the point where their associated path item is encountered. Import path hooks are registered by adding new callables to :data:`sys.path_hooks` as described below. The meta path ------------- .. index:: single: sys.meta_path pair: finder; find_spec When the named module is not found in :data:`sys.modules`, Python next searches :data:`sys.meta_path`, which contains a list of meta path finder objects. These finders are queried in order to see if they know how to handle the named module. Meta path finders must implement a method called :meth:`~importlib.abc.MetaPathFinder.find_spec()` which takes three arguments: a name, an import path, and (optionally) a target module. The meta path finder can use any strategy it wants to determine whether it can handle the named module or not. If the meta path finder knows how to handle the named module, it returns a spec object. If it cannot handle the named module, it returns ``None``. If :data:`sys.meta_path` processing reaches the end of its list without returning a spec, then a :exc:`ModuleNotFoundError` is raised. Any other exceptions raised are simply propagated up, aborting the import process. The :meth:`~importlib.abc.MetaPathFinder.find_spec()` method of meta path finders is called with two or three arguments. The first is the fully qualified name of the module being imported, for example ``foo.bar.baz``. The second argument is the path entries to use for the module search. For top-level modules, the second argument is ``None``, but for submodules or subpackages, the second argument is the value of the parent package's ``__path__`` attribute. If the appropriate ``__path__`` attribute cannot be accessed, a :exc:`ModuleNotFoundError` is raised. The third argument is an existing module object that will be the target of loading later. The import system passes in a target module only during reload. The meta path may be traversed multiple times for a single import request. For example, assuming none of the modules involved has already been cached, importing ``foo.bar.baz`` will first perform a top level import, calling ``mpf.find_spec("foo", None, None)`` on each meta path finder (``mpf``). After ``foo`` has been imported, ``foo.bar`` will be imported by traversing the meta path a second time, calling ``mpf.find_spec("foo.bar", foo.__path__, None)``. Once ``foo.bar`` has been imported, the final traversal will call ``mpf.find_spec("foo.bar.baz", foo.bar.__path__, None)``. Some meta path finders only support top level imports. These importers will always return ``None`` when anything other than ``None`` is passed as the second argument. Python's default :data:`sys.meta_path` has three meta path finders, one that knows how to import built-in modules, one that knows how to import frozen modules, and one that knows how to import modules from an :term:`import path` (i.e. the :term:`path based finder`). .. versionchanged:: 3.4 The :meth:`~importlib.abc.MetaPathFinder.find_spec` method of meta path finders replaced :meth:`~importlib.abc.MetaPathFinder.find_module`, which is now deprecated. While it will continue to work without change, the import machinery will try it only if the finder does not implement ``find_spec()``. Loading ======= If and when a module spec is found, the import machinery will use it (and the loader it contains) when loading the module. Here is an approximation of what happens during the loading portion of import:: module = None if spec.loader is not None and hasattr(spec.loader, 'create_module'): # It is assumed 'exec_module' will also be defined on the loader. module = spec.loader.create_module(spec) if module is None: module = ModuleType(spec.name) # The import-related module attributes get set here: _init_module_attrs(spec, module) if spec.loader is None: if spec.submodule_search_locations is not None: # namespace package sys.modules[spec.name] = module else: # unsupported raise ImportError elif not hasattr(spec.loader, 'exec_module'): module = spec.loader.load_module(spec.name) # Set __loader__ and __package__ if missing. else: sys.modules[spec.name] = module try: spec.loader.exec_module(module) except BaseException: try: del sys.modules[spec.name] except KeyError: pass raise return sys.modules[spec.name] Note the following details: * If there is an existing module object with the given name in :data:`sys.modules`, import will have already returned it. * The module will exist in :data:`sys.modules` before the loader executes the module code. This is crucial because the module code may (directly or indirectly) import itself; adding it to :data:`sys.modules` beforehand prevents unbounded recursion in the worst case and multiple loading in the best. * If loading fails, the failing module -- and only the failing module -- gets removed from :data:`sys.modules`. Any module already in the :data:`sys.modules` cache, and any module that was successfully loaded as a side-effect, must remain in the cache. This contrasts with reloading where even the failing module is left in :data:`sys.modules`. * After the module is created but before execution, the import machinery sets the import-related module attributes ("_init_module_attrs" in the pseudo-code example above), as summarized in a :ref:`later section `. * Module execution is the key moment of loading in which the module's namespace gets populated. Execution is entirely delegated to the loader, which gets to decide what gets populated and how. * The module created during loading and passed to exec_module() may not be the one returned at the end of import [#fnlo]_. .. versionchanged:: 3.4 The import system has taken over the boilerplate responsibilities of loaders. These were previously performed by the :meth:`importlib.abc.Loader.load_module` method. Loaders ------- Module loaders provide the critical function of loading: module execution. The import machinery calls the :meth:`importlib.abc.Loader.exec_module` method with a single argument, the module object to execute. Any value returned from :meth:`~importlib.abc.Loader.exec_module` is ignored. Loaders must satisfy the following requirements: * If the module is a Python module (as opposed to a built-in module or a dynamically loaded extension), the loader should execute the module's code in the module's global name space (``module.__dict__``). * If the loader cannot execute the module, it should raise an :exc:`ImportError`, although any other exception raised during :meth:`~importlib.abc.Loader.exec_module` will be propagated. In many cases, the finder and loader can be the same object; in such cases the :meth:`~importlib.abc.MetaPathFinder.find_spec` method would just return a spec with the loader set to ``self``. Module loaders may opt in to creating the module object during loading by implementing a :meth:`~importlib.abc.Loader.create_module` method. It takes one argument, the module spec, and returns the new module object to use during loading. ``create_module()`` does not need to set any attributes on the module object. If the method returns ``None``, the import machinery will create the new module itself. .. versionadded:: 3.4 The :meth:`~importlib.abc.Loader.create_module` method of loaders. .. versionchanged:: 3.4 The :meth:`~importlib.abc.Loader.load_module` method was replaced by :meth:`~importlib.abc.Loader.exec_module` and the import machinery assumed all the boilerplate responsibilities of loading. For compatibility with existing loaders, the import machinery will use the ``load_module()`` method of loaders if it exists and the loader does not also implement ``exec_module()``. However, ``load_module()`` has been deprecated and loaders should implement ``exec_module()`` instead. The ``load_module()`` method must implement all the boilerplate loading functionality described above in addition to executing the module. All the same constraints apply, with some additional clarification: * If there is an existing module object with the given name in :data:`sys.modules`, the loader must use that existing module. (Otherwise, :func:`importlib.reload` will not work correctly.) If the named module does not exist in :data:`sys.modules`, the loader must create a new module object and add it to :data:`sys.modules`. * The module *must* exist in :data:`sys.modules` before the loader executes the module code, to prevent unbounded recursion or multiple loading. * If loading fails, the loader must remove any modules it has inserted into :data:`sys.modules`, but it must remove **only** the failing module(s), and only if the loader itself has loaded the module(s) explicitly. .. versionchanged:: 3.5 A :exc:`DeprecationWarning` is raised when ``exec_module()`` is defined but ``create_module()`` is not. .. versionchanged:: 3.6 An :exc:`ImportError` is raised when ``exec_module()`` is defined but ``create_module()`` is not. Submodules ---------- When a submodule is loaded using any mechanism (e.g. ``importlib`` APIs, the ``import`` or ``import-from`` statements, or built-in ``__import__()``) a binding is placed in the parent module's namespace to the submodule object. For example, if package ``spam`` has a submodule ``foo``, after importing ``spam.foo``, ``spam`` will have an attribute ``foo`` which is bound to the submodule. Let's say you have the following directory structure:: spam/ __init__.py foo.py bar.py and ``spam/__init__.py`` has the following lines in it:: from .foo import Foo from .bar import Bar then executing the following puts a name binding to ``foo`` and ``bar`` in the ``spam`` module:: >>> import spam >>> spam.foo >>> spam.bar Given Python's familiar name binding rules this might seem surprising, but it's actually a fundamental feature of the import system. The invariant holding is that if you have ``sys.modules['spam']`` and ``sys.modules['spam.foo']`` (as you would after the above import), the latter must appear as the ``foo`` attribute of the former. Module spec ----------- The import machinery uses a variety of information about each module during import, especially before loading. Most of the information is common to all modules. The purpose of a module's spec is to encapsulate this import-related information on a per-module basis. Using a spec during import allows state to be transferred between import system components, e.g. between the finder that creates the module spec and the loader that executes it. Most importantly, it allows the import machinery to perform the boilerplate operations of loading, whereas without a module spec the loader had that responsibility. The module's spec is exposed as the ``__spec__`` attribute on a module object. See :class:`~importlib.machinery.ModuleSpec` for details on the contents of the module spec. .. versionadded:: 3.4 .. _import-mod-attrs: Import-related module attributes -------------------------------- The import machinery fills in these attributes on each module object during loading, based on the module's spec, before the loader executes the module. .. attribute:: __name__ The ``__name__`` attribute must be set to the fully-qualified name of the module. This name is used to uniquely identify the module in the import system. .. attribute:: __loader__ The ``__loader__`` attribute must be set to the loader object that the import machinery used when loading the module. This is mostly for introspection, but can be used for additional loader-specific functionality, for example getting data associated with a loader. .. attribute:: __package__ The module's ``__package__`` attribute must be set. Its value must be a string, but it can be the same value as its ``__name__``. When the module is a package, its ``__package__`` value should be set to its ``__name__``. When the module is not a package, ``__package__`` should be set to the empty string for top-level modules, or for submodules, to the parent package's name. See :pep:`366` for further details. This attribute is used instead of ``__name__`` to calculate explicit relative imports for main modules, as defined in :pep:`366`. It is expected to have the same value as ``__spec__.parent``. .. versionchanged:: 3.6 The value of ``__package__`` is expected to be the same as ``__spec__.parent``. .. attribute:: __spec__ The ``__spec__`` attribute must be set to the module spec that was used when importing the module. Setting ``__spec__`` appropriately applies equally to :ref:`modules initialized during interpreter startup `. The one exception is ``__main__``, where ``__spec__`` is :ref:`set to None in some cases `. When ``__package__`` is not defined, ``__spec__.parent`` is used as a fallback. .. versionadded:: 3.4 .. versionchanged:: 3.6 ``__spec__.parent`` is used as a fallback when ``__package__`` is not defined. .. attribute:: __path__ If the module is a package (either regular or namespace), the module object's ``__path__`` attribute must be set. The value must be iterable, but may be empty if ``__path__`` has no further significance. If ``__path__`` is not empty, it must produce strings when iterated over. More details on the semantics of ``__path__`` are given :ref:`below `. Non-package modules should not have a ``__path__`` attribute. .. attribute:: __file__ .. attribute:: __cached__ ``__file__`` is optional. If set, this attribute's value must be a string. The import system may opt to leave ``__file__`` unset if it has no semantic meaning (e.g. a module loaded from a database). If ``__file__`` is set, it may also be appropriate to set the ``__cached__`` attribute which is the path to any compiled version of the code (e.g. byte-compiled file). The file does not need to exist to set this attribute; the path can simply point to where the compiled file would exist (see :pep:`3147`). It is also appropriate to set ``__cached__`` when ``__file__`` is not set. However, that scenario is quite atypical. Ultimately, the loader is what makes use of ``__file__`` and/or ``__cached__``. So if a loader can load from a cached module but otherwise does not load from a file, that atypical scenario may be appropriate. .. _package-path-rules: module.__path__ --------------- By definition, if a module has a ``__path__`` attribute, it is a package. A package's ``__path__`` attribute is used during imports of its subpackages. Within the import machinery, it functions much the same as :data:`sys.path`, i.e. providing a list of locations to search for modules during import. However, ``__path__`` is typically much more constrained than :data:`sys.path`. ``__path__`` must be an iterable of strings, but it may be empty. The same rules used for :data:`sys.path` also apply to a package's ``__path__``, and :data:`sys.path_hooks` (described below) are consulted when traversing a package's ``__path__``. A package's ``__init__.py`` file may set or alter the package's ``__path__`` attribute, and this was typically the way namespace packages were implemented prior to :pep:`420`. With the adoption of :pep:`420`, namespace packages no longer need to supply ``__init__.py`` files containing only ``__path__`` manipulation code; the import machinery automatically sets ``__path__`` correctly for the namespace package. Module reprs ------------ By default, all modules have a usable repr, however depending on the attributes set above, and in the module's spec, you can more explicitly control the repr of module objects. If the module has a spec (``__spec__``), the import machinery will try to generate a repr from it. If that fails or there is no spec, the import system will craft a default repr using whatever information is available on the module. It will try to use the ``module.__name__``, ``module.__file__``, and ``module.__loader__`` as input into the repr, with defaults for whatever information is missing. Here are the exact rules used: * If the module has a ``__spec__`` attribute, the information in the spec is used to generate the repr. The "name", "loader", "origin", and "has_location" attributes are consulted. * If the module has a ``__file__`` attribute, this is used as part of the module's repr. * If the module has no ``__file__`` but does have a ``__loader__`` that is not ``None``, then the loader's repr is used as part of the module's repr. * Otherwise, just use the module's ``__name__`` in the repr. .. versionchanged:: 3.4 Use of :meth:`loader.module_repr() ` has been deprecated and the module spec is now used by the import machinery to generate a module repr. For backward compatibility with Python 3.3, the module repr will be generated by calling the loader's :meth:`~importlib.abc.Loader.module_repr` method, if defined, before trying either approach described above. However, the method is deprecated. The Path Based Finder ===================== .. index:: single: path based finder As mentioned previously, Python comes with several default meta path finders. One of these, called the :term:`path based finder` (:class:`~importlib.machinery.PathFinder`), searches an :term:`import path`, which contains a list of :term:`path entries `. Each path entry names a location to search for modules. The path based finder itself doesn't know how to import anything. Instead, it traverses the individual path entries, associating each of them with a path entry finder that knows how to handle that particular kind of path. The default set of path entry finders implement all the semantics for finding modules on the file system, handling special file types such as Python source code (``.py`` files), Python byte code (``.pyc`` files) and shared libraries (e.g. ``.so`` files). When supported by the :mod:`zipimport` module in the standard library, the default path entry finders also handle loading all of these file types (other than shared libraries) from zipfiles. Path entries need not be limited to file system locations. They can refer to URLs, database queries, or any other location that can be specified as a string. The path based finder provides additional hooks and protocols so that you can extend and customize the types of searchable path entries. For example, if you wanted to support path entries as network URLs, you could write a hook that implements HTTP semantics to find modules on the web. This hook (a callable) would return a :term:`path entry finder` supporting the protocol described below, which was then used to get a loader for the module from the web. A word of warning: this section and the previous both use the term *finder*, distinguishing between them by using the terms :term:`meta path finder` and :term:`path entry finder`. These two types of finders are very similar, support similar protocols, and function in similar ways during the import process, but it's important to keep in mind that they are subtly different. In particular, meta path finders operate at the beginning of the import process, as keyed off the :data:`sys.meta_path` traversal. By contrast, path entry finders are in a sense an implementation detail of the path based finder, and in fact, if the path based finder were to be removed from :data:`sys.meta_path`, none of the path entry finder semantics would be invoked. Path entry finders ------------------ .. index:: single: sys.path single: sys.path_hooks single: sys.path_importer_cache single: PYTHONPATH The :term:`path based finder` is responsible for finding and loading Python modules and packages whose location is specified with a string :term:`path entry`. Most path entries name locations in the file system, but they need not be limited to this. As a meta path finder, the :term:`path based finder` implements the :meth:`~importlib.abc.MetaPathFinder.find_spec` protocol previously described, however it exposes additional hooks that can be used to customize how modules are found and loaded from the :term:`import path`. Three variables are used by the :term:`path based finder`, :data:`sys.path`, :data:`sys.path_hooks` and :data:`sys.path_importer_cache`. The ``__path__`` attributes on package objects are also used. These provide additional ways that the import machinery can be customized. :data:`sys.path` contains a list of strings providing search locations for modules and packages. It is initialized from the :data:`PYTHONPATH` environment variable and various other installation- and implementation-specific defaults. Entries in :data:`sys.path` can name directories on the file system, zip files, and potentially other "locations" (see the :mod:`site` module) that should be searched for modules, such as URLs, or database queries. Only strings and bytes should be present on :data:`sys.path`; all other data types are ignored. The encoding of bytes entries is determined by the individual :term:`path entry finders `. The :term:`path based finder` is a :term:`meta path finder`, so the import machinery begins the :term:`import path` search by calling the path based finder's :meth:`~importlib.machinery.PathFinder.find_spec` method as described previously. When the ``path`` argument to :meth:`~importlib.machinery.PathFinder.find_spec` is given, it will be a list of string paths to traverse - typically a package's ``__path__`` attribute for an import within that package. If the ``path`` argument is ``None``, this indicates a top level import and :data:`sys.path` is used. The path based finder iterates over every entry in the search path, and for each of these, looks for an appropriate :term:`path entry finder` (:class:`~importlib.abc.PathEntryFinder`) for the path entry. Because this can be an expensive operation (e.g. there may be `stat()` call overheads for this search), the path based finder maintains a cache mapping path entries to path entry finders. This cache is maintained in :data:`sys.path_importer_cache` (despite the name, this cache actually stores finder objects rather than being limited to :term:`importer` objects). In this way, the expensive search for a particular :term:`path entry` location's :term:`path entry finder` need only be done once. User code is free to remove cache entries from :data:`sys.path_importer_cache` forcing the path based finder to perform the path entry search again [#fnpic]_. If the path entry is not present in the cache, the path based finder iterates over every callable in :data:`sys.path_hooks`. Each of the :term:`path entry hooks ` in this list is called with a single argument, the path entry to be searched. This callable may either return a :term:`path entry finder` that can handle the path entry, or it may raise :exc:`ImportError`. An :exc:`ImportError` is used by the path based finder to signal that the hook cannot find a :term:`path entry finder` for that :term:`path entry`. The exception is ignored and :term:`import path` iteration continues. The hook should expect either a string or bytes object; the encoding of bytes objects is up to the hook (e.g. it may be a file system encoding, UTF-8, or something else), and if the hook cannot decode the argument, it should raise :exc:`ImportError`. If :data:`sys.path_hooks` iteration ends with no :term:`path entry finder` being returned, then the path based finder's :meth:`~importlib.machinery.PathFinder.find_spec` method will store ``None`` in :data:`sys.path_importer_cache` (to indicate that there is no finder for this path entry) and return ``None``, indicating that this :term:`meta path finder` could not find the module. If a :term:`path entry finder` *is* returned by one of the :term:`path entry hook` callables on :data:`sys.path_hooks`, then the following protocol is used to ask the finder for a module spec, which is then used when loading the module. The current working directory -- denoted by an empty string -- is handled slightly differently from other entries on :data:`sys.path`. First, if the current working directory is found to not exist, no value is stored in :data:`sys.path_importer_cache`. Second, the value for the current working directory is looked up fresh for each module lookup. Third, the path used for :data:`sys.path_importer_cache` and returned by :meth:`importlib.machinery.PathFinder.find_spec` will be the actual current working directory and not the empty string. Path entry finder protocol -------------------------- In order to support imports of modules and initialized packages and also to contribute portions to namespace packages, path entry finders must implement the :meth:`~importlib.abc.PathEntryFinder.find_spec` method. :meth:`~importlib.abc.PathEntryFinder.find_spec` takes two argument, the fully qualified name of the module being imported, and the (optional) target module. ``find_spec()`` returns a fully populated spec for the module. This spec will always have "loader" set (with one exception). To indicate to the import machinery that the spec represents a namespace :term:`portion`. the path entry finder sets "loader" on the spec to ``None`` and "submodule_search_locations" to a list containing the portion. .. versionchanged:: 3.4 :meth:`~importlib.abc.PathEntryFinder.find_spec` replaced :meth:`~importlib.abc.PathEntryFinder.find_loader` and :meth:`~importlib.abc.PathEntryFinder.find_module`, both of which are now deprecated, but will be used if ``find_spec()`` is not defined. Older path entry finders may implement one of these two deprecated methods instead of ``find_spec()``. The methods are still respected for the sake of backward compatibility. However, if ``find_spec()`` is implemented on the path entry finder, the legacy methods are ignored. :meth:`~importlib.abc.PathEntryFinder.find_loader` takes one argument, the fully qualified name of the module being imported. ``find_loader()`` returns a 2-tuple where the first item is the loader and the second item is a namespace :term:`portion`. When the first item (i.e. the loader) is ``None``, this means that while the path entry finder does not have a loader for the named module, it knows that the path entry contributes to a namespace portion for the named module. This will almost always be the case where Python is asked to import a namespace package that has no physical presence on the file system. When a path entry finder returns ``None`` for the loader, the second item of the 2-tuple return value must be a sequence, although it can be empty. If ``find_loader()`` returns a non-``None`` loader value, the portion is ignored and the loader is returned from the path based finder, terminating the search through the path entries. For backwards compatibility with other implementations of the import protocol, many path entry finders also support the same, traditional ``find_module()`` method that meta path finders support. However path entry finder ``find_module()`` methods are never called with a ``path`` argument (they are expected to record the appropriate path information from the initial call to the path hook). The ``find_module()`` method on path entry finders is deprecated, as it does not allow the path entry finder to contribute portions to namespace packages. If both ``find_loader()`` and ``find_module()`` exist on a path entry finder, the import system will always call ``find_loader()`` in preference to ``find_module()``. Replacing the standard import system ==================================== The most reliable mechanism for replacing the entire import system is to delete the default contents of :data:`sys.meta_path`, replacing them entirely with a custom meta path hook. If it is acceptable to only alter the behaviour of import statements without affecting other APIs that access the import system, then replacing the builtin :func:`__import__` function may be sufficient. This technique may also be employed at the module level to only alter the behaviour of import statements within that module. To selectively prevent import of some modules from a hook early on the meta path (rather than disabling the standard import system entirely), it is sufficient to raise :exc:`ModuleNotFoundError` directly from :meth:`~importlib.abc.MetaPathFinder.find_spec` instead of returning ``None``. The latter indicates that the meta path search should continue, while raising an exception terminates it immediately. Special considerations for __main__ =================================== The :mod:`__main__` module is a special case relative to Python's import system. As noted :ref:`elsewhere `, the ``__main__`` module is directly initialized at interpreter startup, much like :mod:`sys` and :mod:`builtins`. However, unlike those two, it doesn't strictly qualify as a built-in module. This is because the manner in which ``__main__`` is initialized depends on the flags and other options with which the interpreter is invoked. .. _main_spec: __main__.__spec__ ----------------- Depending on how :mod:`__main__` is initialized, ``__main__.__spec__`` gets set appropriately or to ``None``. When Python is started with the :option:`-m` option, ``__spec__`` is set to the module spec of the corresponding module or package. ``__spec__`` is also populated when the ``__main__`` module is loaded as part of executing a directory, zipfile or other :data:`sys.path` entry. In :ref:`the remaining cases ` ``__main__.__spec__`` is set to ``None``, as the code used to populate the :mod:`__main__` does not correspond directly with an importable module: - interactive prompt - -c switch - running from stdin - running directly from a source or bytecode file Note that ``__main__.__spec__`` is always ``None`` in the last case, *even if* the file could technically be imported directly as a module instead. Use the :option:`-m` switch if valid module metadata is desired in :mod:`__main__`. Note also that even when ``__main__`` corresponds with an importable module and ``__main__.__spec__`` is set accordingly, they're still considered *distinct* modules. This is due to the fact that blocks guarded by ``if __name__ == "__main__":`` checks only execute when the module is used to populate the ``__main__`` namespace, and not during normal import. Open issues =========== XXX It would be really nice to have a diagram. XXX * (import_machinery.rst) how about a section devoted just to the attributes of modules and packages, perhaps expanding upon or supplanting the related entries in the data model reference page? XXX runpy, pkgutil, et al in the library manual should all get "See Also" links at the top pointing to the new import system section. XXX Add more explanation regarding the different ways in which ``__main__`` is initialized? XXX Add more info on ``__main__`` quirks/pitfalls (i.e. copy from :pep:`395`). References ========== The import machinery has evolved considerably since Python's early days. The original `specification for packages `_ is still available to read, although some details have changed since the writing of that document. The original specification for :data:`sys.meta_path` was :pep:`302`, with subsequent extension in :pep:`420`. :pep:`420` introduced :term:`namespace packages ` for Python 3.3. :pep:`420` also introduced the :meth:`find_loader` protocol as an alternative to :meth:`find_module`. :pep:`366` describes the addition of the ``__package__`` attribute for explicit relative imports in main modules. :pep:`328` introduced absolute and explicit relative imports and initially proposed ``__name__`` for semantics :pep:`366` would eventually specify for ``__package__``. :pep:`338` defines executing modules as scripts. :pep:`451` adds the encapsulation of per-module import state in spec objects. It also off-loads most of the boilerplate responsibilities of loaders back onto the import machinery. These changes allow the deprecation of several APIs in the import system and also addition of new methods to finders and loaders. .. rubric:: Footnotes .. [#fnmo] See :class:`types.ModuleType`. .. [#fnlo] The importlib implementation avoids using the return value directly. Instead, it gets the module object by looking the module name up in :data:`sys.modules`. The indirect effect of this is that an imported module may replace itself in :data:`sys.modules`. This is implementation-specific behavior that is not guaranteed to work in other Python implementations. .. [#fnpic] In legacy code, it is possible to find instances of :class:`imp.NullImporter` in the :data:`sys.path_importer_cache`. It is recommended that code be changed to use ``None`` instead. See :ref:`portingpythoncode` for more details. PK 3][[reference/introduction.rst.txtnu[ .. _introduction: ************ Introduction ************ This reference manual describes the Python programming language. It is not intended as a tutorial. While I am trying to be as precise as possible, I chose to use English rather than formal specifications for everything except syntax and lexical analysis. This should make the document more understandable to the average reader, but will leave room for ambiguities. Consequently, if you were coming from Mars and tried to re-implement Python from this document alone, you might have to guess things and in fact you would probably end up implementing quite a different language. On the other hand, if you are using Python and wonder what the precise rules about a particular area of the language are, you should definitely be able to find them here. If you would like to see a more formal definition of the language, maybe you could volunteer your time --- or invent a cloning machine :-). It is dangerous to add too many implementation details to a language reference document --- the implementation may change, and other implementations of the same language may work differently. On the other hand, CPython is the one Python implementation in widespread use (although alternate implementations continue to gain support), and its particular quirks are sometimes worth being mentioned, especially where the implementation imposes additional limitations. Therefore, you'll find short "implementation notes" sprinkled throughout the text. Every Python implementation comes with a number of built-in and standard modules. These are documented in :ref:`library-index`. A few built-in modules are mentioned when they interact in a significant way with the language definition. .. _implementations: Alternate Implementations ========================= Though there is one Python implementation which is by far the most popular, there are some alternate implementations which are of particular interest to different audiences. Known implementations include: CPython This is the original and most-maintained implementation of Python, written in C. New language features generally appear here first. Jython Python implemented in Java. This implementation can be used as a scripting language for Java applications, or can be used to create applications using the Java class libraries. It is also often used to create tests for Java libraries. More information can be found at `the Jython website `_. Python for .NET This implementation actually uses the CPython implementation, but is a managed .NET application and makes .NET libraries available. It was created by Brian Lloyd. For more information, see the `Python for .NET home page `_. IronPython An alternate Python for .NET. Unlike Python.NET, this is a complete Python implementation that generates IL, and compiles Python code directly to .NET assemblies. It was created by Jim Hugunin, the original creator of Jython. For more information, see `the IronPython website `_. PyPy An implementation of Python written completely in Python. It supports several advanced features not found in other implementations like stackless support and a Just in Time compiler. One of the goals of the project is to encourage experimentation with the language itself by making it easier to modify the interpreter (since it is written in Python). Additional information is available on `the PyPy project's home page `_. Each of these implementations varies in some way from the language as documented in this manual, or introduces specific information beyond what's covered in the standard Python documentation. Please refer to the implementation-specific documentation to determine what else you need to know about the specific implementation you're using. .. _notation: Notation ======== .. index:: BNF, grammar, syntax, notation The descriptions of lexical analysis and syntax use a modified BNF grammar notation. This uses the following style of definition: .. productionlist:: * name: `lc_letter` (`lc_letter` | "_")* lc_letter: "a"..."z" The first line says that a ``name`` is an ``lc_letter`` followed by a sequence of zero or more ``lc_letter``\ s and underscores. An ``lc_letter`` in turn is any of the single characters ``'a'`` through ``'z'``. (This rule is actually adhered to for the names defined in lexical and grammar rules in this document.) Each rule begins with a name (which is the name defined by the rule) and ``::=``. A vertical bar (``|``) is used to separate alternatives; it is the least binding operator in this notation. A star (``*``) means zero or more repetitions of the preceding item; likewise, a plus (``+``) means one or more repetitions, and a phrase enclosed in square brackets (``[ ]``) means zero or one occurrences (in other words, the enclosed phrase is optional). The ``*`` and ``+`` operators bind as tightly as possible; parentheses are used for grouping. Literal strings are enclosed in quotes. White space is only meaningful to separate tokens. Rules are normally contained on a single line; rules with many alternatives may be formatted alternatively with each line after the first beginning with a vertical bar. .. index:: lexical definitions, ASCII In lexical definitions (as the example above), two more conventions are used: Two literal characters separated by three dots mean a choice of any single character in the given (inclusive) range of ASCII characters. A phrase between angular brackets (``<...>``) gives an informal description of the symbol defined; e.g., this could be used to describe the notion of 'control character' if needed. Even though the notation used is almost the same, there is a big difference between the meaning of lexical and syntactic definitions: a lexical definition operates on the individual characters of the input source, while a syntax definition operates on the stream of tokens generated by the lexical analysis. All uses of BNF in the next chapter ("Lexical Analysis") are lexical definitions; uses in subsequent chapters are syntactic definitions. PK 3]bA''reference/datamodel.rst.txtnu[ .. _datamodel: ********** Data model ********** .. _objects: Objects, values and types ========================= .. index:: single: object single: data :dfn:`Objects` are Python's abstraction for data. All data in a Python program is represented by objects or by relations between objects. (In a sense, and in conformance to Von Neumann's model of a "stored program computer," code is also represented by objects.) .. index:: builtin: id builtin: type single: identity of an object single: value of an object single: type of an object single: mutable object single: immutable object .. XXX it *is* now possible in some cases to change an object's type, under certain controlled conditions Every object has an identity, a type and a value. An object's *identity* never changes once it has been created; you may think of it as the object's address in memory. The ':keyword:`is`' operator compares the identity of two objects; the :func:`id` function returns an integer representing its identity. .. impl-detail:: For CPython, ``id(x)`` is the memory address where ``x`` is stored. An object's type determines the operations that the object supports (e.g., "does it have a length?") and also defines the possible values for objects of that type. The :func:`type` function returns an object's type (which is an object itself). Like its identity, an object's :dfn:`type` is also unchangeable. [#]_ The *value* of some objects can change. Objects whose value can change are said to be *mutable*; objects whose value is unchangeable once they are created are called *immutable*. (The value of an immutable container object that contains a reference to a mutable object can change when the latter's value is changed; however the container is still considered immutable, because the collection of objects it contains cannot be changed. So, immutability is not strictly the same as having an unchangeable value, it is more subtle.) An object's mutability is determined by its type; for instance, numbers, strings and tuples are immutable, while dictionaries and lists are mutable. .. index:: single: garbage collection single: reference counting single: unreachable object Objects are never explicitly destroyed; however, when they become unreachable they may be garbage-collected. An implementation is allowed to postpone garbage collection or omit it altogether --- it is a matter of implementation quality how garbage collection is implemented, as long as no objects are collected that are still reachable. .. impl-detail:: CPython currently uses a reference-counting scheme with (optional) delayed detection of cyclically linked garbage, which collects most objects as soon as they become unreachable, but is not guaranteed to collect garbage containing circular references. See the documentation of the :mod:`gc` module for information on controlling the collection of cyclic garbage. Other implementations act differently and CPython may change. Do not depend on immediate finalization of objects when they become unreachable (so you should always close files explicitly). Note that the use of the implementation's tracing or debugging facilities may keep objects alive that would normally be collectable. Also note that catching an exception with a ':keyword:`try`...\ :keyword:`except`' statement may keep objects alive. Some objects contain references to "external" resources such as open files or windows. It is understood that these resources are freed when the object is garbage-collected, but since garbage collection is not guaranteed to happen, such objects also provide an explicit way to release the external resource, usually a :meth:`close` method. Programs are strongly recommended to explicitly close such objects. The ':keyword:`try`...\ :keyword:`finally`' statement and the ':keyword:`with`' statement provide convenient ways to do this. .. index:: single: container Some objects contain references to other objects; these are called *containers*. Examples of containers are tuples, lists and dictionaries. The references are part of a container's value. In most cases, when we talk about the value of a container, we imply the values, not the identities of the contained objects; however, when we talk about the mutability of a container, only the identities of the immediately contained objects are implied. So, if an immutable container (like a tuple) contains a reference to a mutable object, its value changes if that mutable object is changed. Types affect almost all aspects of object behavior. Even the importance of object identity is affected in some sense: for immutable types, operations that compute new values may actually return a reference to any existing object with the same type and value, while for mutable objects this is not allowed. E.g., after ``a = 1; b = 1``, ``a`` and ``b`` may or may not refer to the same object with the value one, depending on the implementation, but after ``c = []; d = []``, ``c`` and ``d`` are guaranteed to refer to two different, unique, newly created empty lists. (Note that ``c = d = []`` assigns the same object to both ``c`` and ``d``.) .. _types: The standard type hierarchy =========================== .. index:: single: type pair: data; type pair: type; hierarchy pair: extension; module pair: C; language Below is a list of the types that are built into Python. Extension modules (written in C, Java, or other languages, depending on the implementation) can define additional types. Future versions of Python may add types to the type hierarchy (e.g., rational numbers, efficiently stored arrays of integers, etc.), although such additions will often be provided via the standard library instead. .. index:: single: attribute pair: special; attribute triple: generic; special; attribute Some of the type descriptions below contain a paragraph listing 'special attributes.' These are attributes that provide access to the implementation and are not intended for general use. Their definition may change in the future. None .. index:: object: None This type has a single value. There is a single object with this value. This object is accessed through the built-in name ``None``. It is used to signify the absence of a value in many situations, e.g., it is returned from functions that don't explicitly return anything. Its truth value is false. NotImplemented .. index:: object: NotImplemented This type has a single value. There is a single object with this value. This object is accessed through the built-in name ``NotImplemented``. Numeric methods and rich comparison methods should return this value if they do not implement the operation for the operands provided. (The interpreter will then try the reflected operation, or some other fallback, depending on the operator.) Its truth value is true. See :ref:`implementing-the-arithmetic-operations` for more details. Ellipsis .. index:: object: Ellipsis This type has a single value. There is a single object with this value. This object is accessed through the literal ``...`` or the built-in name ``Ellipsis``. Its truth value is true. :class:`numbers.Number` .. index:: object: numeric These are created by numeric literals and returned as results by arithmetic operators and arithmetic built-in functions. Numeric objects are immutable; once created their value never changes. Python numbers are of course strongly related to mathematical numbers, but subject to the limitations of numerical representation in computers. Python distinguishes between integers, floating point numbers, and complex numbers: :class:`numbers.Integral` .. index:: object: integer These represent elements from the mathematical set of integers (positive and negative). There are two types of integers: Integers (:class:`int`) These represent numbers in an unlimited range, subject to available (virtual) memory only. For the purpose of shift and mask operations, a binary representation is assumed, and negative numbers are represented in a variant of 2's complement which gives the illusion of an infinite string of sign bits extending to the left. Booleans (:class:`bool`) .. index:: object: Boolean single: False single: True These represent the truth values False and True. The two objects representing the values ``False`` and ``True`` are the only Boolean objects. The Boolean type is a subtype of the integer type, and Boolean values behave like the values 0 and 1, respectively, in almost all contexts, the exception being that when converted to a string, the strings ``"False"`` or ``"True"`` are returned, respectively. .. index:: pair: integer; representation The rules for integer representation are intended to give the most meaningful interpretation of shift and mask operations involving negative integers. :class:`numbers.Real` (:class:`float`) .. index:: object: floating point pair: floating point; number pair: C; language pair: Java; language These represent machine-level double precision floating point numbers. You are at the mercy of the underlying machine architecture (and C or Java implementation) for the accepted range and handling of overflow. Python does not support single-precision floating point numbers; the savings in processor and memory usage that are usually the reason for using these are dwarfed by the overhead of using objects in Python, so there is no reason to complicate the language with two kinds of floating point numbers. :class:`numbers.Complex` (:class:`complex`) .. index:: object: complex pair: complex; number These represent complex numbers as a pair of machine-level double precision floating point numbers. The same caveats apply as for floating point numbers. The real and imaginary parts of a complex number ``z`` can be retrieved through the read-only attributes ``z.real`` and ``z.imag``. Sequences .. index:: builtin: len object: sequence single: index operation single: item selection single: subscription These represent finite ordered sets indexed by non-negative numbers. The built-in function :func:`len` returns the number of items of a sequence. When the length of a sequence is *n*, the index set contains the numbers 0, 1, ..., *n*-1. Item *i* of sequence *a* is selected by ``a[i]``. .. index:: single: slicing Sequences also support slicing: ``a[i:j]`` selects all items with index *k* such that *i* ``<=`` *k* ``<`` *j*. When used as an expression, a slice is a sequence of the same type. This implies that the index set is renumbered so that it starts at 0. Some sequences also support "extended slicing" with a third "step" parameter: ``a[i:j:k]`` selects all items of *a* with index *x* where ``x = i + n*k``, *n* ``>=`` ``0`` and *i* ``<=`` *x* ``<`` *j*. Sequences are distinguished according to their mutability: Immutable sequences .. index:: object: immutable sequence object: immutable An object of an immutable sequence type cannot change once it is created. (If the object contains references to other objects, these other objects may be mutable and may be changed; however, the collection of objects directly referenced by an immutable object cannot change.) The following types are immutable sequences: .. index:: single: string; immutable sequences Strings .. index:: builtin: chr builtin: ord single: character single: integer single: Unicode A string is a sequence of values that represent Unicode code points. All the code points in the range ``U+0000 - U+10FFFF`` can be represented in a string. Python doesn't have a :c:type:`char` type; instead, every code point in the string is represented as a string object with length ``1``. The built-in function :func:`ord` converts a code point from its string form to an integer in the range ``0 - 10FFFF``; :func:`chr` converts an integer in the range ``0 - 10FFFF`` to the corresponding length ``1`` string object. :meth:`str.encode` can be used to convert a :class:`str` to :class:`bytes` using the given text encoding, and :meth:`bytes.decode` can be used to achieve the opposite. Tuples .. index:: object: tuple pair: singleton; tuple pair: empty; tuple The items of a tuple are arbitrary Python objects. Tuples of two or more items are formed by comma-separated lists of expressions. A tuple of one item (a 'singleton') can be formed by affixing a comma to an expression (an expression by itself does not create a tuple, since parentheses must be usable for grouping of expressions). An empty tuple can be formed by an empty pair of parentheses. Bytes .. index:: bytes, byte A bytes object is an immutable array. The items are 8-bit bytes, represented by integers in the range 0 <= x < 256. Bytes literals (like ``b'abc'``) and the built-in :func:`bytes()` constructor can be used to create bytes objects. Also, bytes objects can be decoded to strings via the :meth:`~bytes.decode` method. Mutable sequences .. index:: object: mutable sequence object: mutable pair: assignment; statement single: subscription single: slicing Mutable sequences can be changed after they are created. The subscription and slicing notations can be used as the target of assignment and :keyword:`del` (delete) statements. There are currently two intrinsic mutable sequence types: Lists .. index:: object: list The items of a list are arbitrary Python objects. Lists are formed by placing a comma-separated list of expressions in square brackets. (Note that there are no special cases needed to form lists of length 0 or 1.) Byte Arrays .. index:: bytearray A bytearray object is a mutable array. They are created by the built-in :func:`bytearray` constructor. Aside from being mutable (and hence unhashable), byte arrays otherwise provide the same interface and functionality as immutable :class:`bytes` objects. .. index:: module: array The extension module :mod:`array` provides an additional example of a mutable sequence type, as does the :mod:`collections` module. Set types .. index:: builtin: len object: set type These represent unordered, finite sets of unique, immutable objects. As such, they cannot be indexed by any subscript. However, they can be iterated over, and the built-in function :func:`len` returns the number of items in a set. Common uses for sets are fast membership testing, removing duplicates from a sequence, and computing mathematical operations such as intersection, union, difference, and symmetric difference. For set elements, the same immutability rules apply as for dictionary keys. Note that numeric types obey the normal rules for numeric comparison: if two numbers compare equal (e.g., ``1`` and ``1.0``), only one of them can be contained in a set. There are currently two intrinsic set types: Sets .. index:: object: set These represent a mutable set. They are created by the built-in :func:`set` constructor and can be modified afterwards by several methods, such as :meth:`~set.add`. Frozen sets .. index:: object: frozenset These represent an immutable set. They are created by the built-in :func:`frozenset` constructor. As a frozenset is immutable and :term:`hashable`, it can be used again as an element of another set, or as a dictionary key. Mappings .. index:: builtin: len single: subscription object: mapping These represent finite sets of objects indexed by arbitrary index sets. The subscript notation ``a[k]`` selects the item indexed by ``k`` from the mapping ``a``; this can be used in expressions and as the target of assignments or :keyword:`del` statements. The built-in function :func:`len` returns the number of items in a mapping. There is currently a single intrinsic mapping type: Dictionaries .. index:: object: dictionary These represent finite sets of objects indexed by nearly arbitrary values. The only types of values not acceptable as keys are values containing lists or dictionaries or other mutable types that are compared by value rather than by object identity, the reason being that the efficient implementation of dictionaries requires a key's hash value to remain constant. Numeric types used for keys obey the normal rules for numeric comparison: if two numbers compare equal (e.g., ``1`` and ``1.0``) then they can be used interchangeably to index the same dictionary entry. Dictionaries are mutable; they can be created by the ``{...}`` notation (see section :ref:`dict`). .. index:: module: dbm.ndbm module: dbm.gnu The extension modules :mod:`dbm.ndbm` and :mod:`dbm.gnu` provide additional examples of mapping types, as does the :mod:`collections` module. Callable types .. index:: object: callable pair: function; call single: invocation pair: function; argument These are the types to which the function call operation (see section :ref:`calls`) can be applied: User-defined functions .. index:: pair: user-defined; function object: function object: user-defined function A user-defined function object is created by a function definition (see section :ref:`function`). It should be called with an argument list containing the same number of items as the function's formal parameter list. Special attributes: .. tabularcolumns:: |l|L|l| .. index:: single: __doc__ (function attribute) single: __name__ (function attribute) single: __module__ (function attribute) single: __dict__ (function attribute) single: __defaults__ (function attribute) single: __closure__ (function attribute) single: __code__ (function attribute) single: __globals__ (function attribute) single: __annotations__ (function attribute) single: __kwdefaults__ (function attribute) pair: global; namespace +-------------------------+-------------------------------+-----------+ | Attribute | Meaning | | +=========================+===============================+===========+ | :attr:`__doc__` | The function's documentation | Writable | | | string, or ``None`` if | | | | unavailable; not inherited by | | | | subclasses | | +-------------------------+-------------------------------+-----------+ | :attr:`~definition.\ | The function's name | Writable | | __name__` | | | +-------------------------+-------------------------------+-----------+ | :attr:`~definition.\ | The function's | Writable | | __qualname__` | :term:`qualified name` | | | | | | | | .. versionadded:: 3.3 | | +-------------------------+-------------------------------+-----------+ | :attr:`__module__` | The name of the module the | Writable | | | function was defined in, or | | | | ``None`` if unavailable. | | +-------------------------+-------------------------------+-----------+ | :attr:`__defaults__` | A tuple containing default | Writable | | | argument values for those | | | | arguments that have defaults, | | | | or ``None`` if no arguments | | | | have a default value | | +-------------------------+-------------------------------+-----------+ | :attr:`__code__` | The code object representing | Writable | | | the compiled function body. | | +-------------------------+-------------------------------+-----------+ | :attr:`__globals__` | A reference to the dictionary | Read-only | | | that holds the function's | | | | global variables --- the | | | | global namespace of the | | | | module in which the function | | | | was defined. | | +-------------------------+-------------------------------+-----------+ | :attr:`~object.__dict__`| The namespace supporting | Writable | | | arbitrary function | | | | attributes. | | +-------------------------+-------------------------------+-----------+ | :attr:`__closure__` | ``None`` or a tuple of cells | Read-only | | | that contain bindings for the | | | | function's free variables. | | +-------------------------+-------------------------------+-----------+ | :attr:`__annotations__` | A dict containing annotations | Writable | | | of parameters. The keys of | | | | the dict are the parameter | | | | names, and ``'return'`` for | | | | the return annotation, if | | | | provided. | | +-------------------------+-------------------------------+-----------+ | :attr:`__kwdefaults__` | A dict containing defaults | Writable | | | for keyword-only parameters. | | +-------------------------+-------------------------------+-----------+ Most of the attributes labelled "Writable" check the type of the assigned value. Function objects also support getting and setting arbitrary attributes, which can be used, for example, to attach metadata to functions. Regular attribute dot-notation is used to get and set such attributes. *Note that the current implementation only supports function attributes on user-defined functions. Function attributes on built-in functions may be supported in the future.* Additional information about a function's definition can be retrieved from its code object; see the description of internal types below. Instance methods .. index:: object: method object: user-defined method pair: user-defined; method An instance method object combines a class, a class instance and any callable object (normally a user-defined function). .. index:: single: __func__ (method attribute) single: __self__ (method attribute) single: __doc__ (method attribute) single: __name__ (method attribute) single: __module__ (method attribute) Special read-only attributes: :attr:`__self__` is the class instance object, :attr:`__func__` is the function object; :attr:`__doc__` is the method's documentation (same as ``__func__.__doc__``); :attr:`~definition.__name__` is the method name (same as ``__func__.__name__``); :attr:`__module__` is the name of the module the method was defined in, or ``None`` if unavailable. Methods also support accessing (but not setting) the arbitrary function attributes on the underlying function object. User-defined method objects may be created when getting an attribute of a class (perhaps via an instance of that class), if that attribute is a user-defined function object or a class method object. When an instance method object is created by retrieving a user-defined function object from a class via one of its instances, its :attr:`__self__` attribute is the instance, and the method object is said to be bound. The new method's :attr:`__func__` attribute is the original function object. When a user-defined method object is created by retrieving another method object from a class or instance, the behaviour is the same as for a function object, except that the :attr:`__func__` attribute of the new instance is not the original method object but its :attr:`__func__` attribute. When an instance method object is created by retrieving a class method object from a class or instance, its :attr:`__self__` attribute is the class itself, and its :attr:`__func__` attribute is the function object underlying the class method. When an instance method object is called, the underlying function (:attr:`__func__`) is called, inserting the class instance (:attr:`__self__`) in front of the argument list. For instance, when :class:`C` is a class which contains a definition for a function :meth:`f`, and ``x`` is an instance of :class:`C`, calling ``x.f(1)`` is equivalent to calling ``C.f(x, 1)``. When an instance method object is derived from a class method object, the "class instance" stored in :attr:`__self__` will actually be the class itself, so that calling either ``x.f(1)`` or ``C.f(1)`` is equivalent to calling ``f(C,1)`` where ``f`` is the underlying function. Note that the transformation from function object to instance method object happens each time the attribute is retrieved from the instance. In some cases, a fruitful optimization is to assign the attribute to a local variable and call that local variable. Also notice that this transformation only happens for user-defined functions; other callable objects (and all non-callable objects) are retrieved without transformation. It is also important to note that user-defined functions which are attributes of a class instance are not converted to bound methods; this *only* happens when the function is an attribute of the class. Generator functions .. index:: single: generator; function single: generator; iterator A function or method which uses the :keyword:`yield` statement (see section :ref:`yield`) is called a :dfn:`generator function`. Such a function, when called, always returns an iterator object which can be used to execute the body of the function: calling the iterator's :meth:`iterator.__next__` method will cause the function to execute until it provides a value using the :keyword:`yield` statement. When the function executes a :keyword:`return` statement or falls off the end, a :exc:`StopIteration` exception is raised and the iterator will have reached the end of the set of values to be returned. Coroutine functions .. index:: single: coroutine; function A function or method which is defined using :keyword:`async def` is called a :dfn:`coroutine function`. Such a function, when called, returns a :term:`coroutine` object. It may contain :keyword:`await` expressions, as well as :keyword:`async with` and :keyword:`async for` statements. See also the :ref:`coroutine-objects` section. Asynchronous generator functions .. index:: single: asynchronous generator; function single: asynchronous generator; asynchronous iterator A function or method which is defined using :keyword:`async def` and which uses the :keyword:`yield` statement is called a :dfn:`asynchronous generator function`. Such a function, when called, returns an asynchronous iterator object which can be used in an :keyword:`async for` statement to execute the body of the function. Calling the asynchronous iterator's :meth:`aiterator.__anext__` method will return an :term:`awaitable` which when awaited will execute until it provides a value using the :keyword:`yield` expression. When the function executes an empty :keyword:`return` statement or falls off the end, a :exc:`StopAsyncIteration` exception is raised and the asynchronous iterator will have reached the end of the set of values to be yielded. Built-in functions .. index:: object: built-in function object: function pair: C; language A built-in function object is a wrapper around a C function. Examples of built-in functions are :func:`len` and :func:`math.sin` (:mod:`math` is a standard built-in module). The number and type of the arguments are determined by the C function. Special read-only attributes: :attr:`__doc__` is the function's documentation string, or ``None`` if unavailable; :attr:`~definition.__name__` is the function's name; :attr:`__self__` is set to ``None`` (but see the next item); :attr:`__module__` is the name of the module the function was defined in or ``None`` if unavailable. Built-in methods .. index:: object: built-in method object: method pair: built-in; method This is really a different disguise of a built-in function, this time containing an object passed to the C function as an implicit extra argument. An example of a built-in method is ``alist.append()``, assuming *alist* is a list object. In this case, the special read-only attribute :attr:`__self__` is set to the object denoted by *alist*. Classes Classes are callable. These objects normally act as factories for new instances of themselves, but variations are possible for class types that override :meth:`__new__`. The arguments of the call are passed to :meth:`__new__` and, in the typical case, to :meth:`__init__` to initialize the new instance. Class Instances Instances of arbitrary classes can be made callable by defining a :meth:`__call__` method in their class. Modules .. index:: statement: import object: module Modules are a basic organizational unit of Python code, and are created by the :ref:`import system ` as invoked either by the :keyword:`import` statement (see :keyword:`import`), or by calling functions such as :func:`importlib.import_module` and built-in :func:`__import__`. A module object has a namespace implemented by a dictionary object (this is the dictionary referenced by the ``__globals__`` attribute of functions defined in the module). Attribute references are translated to lookups in this dictionary, e.g., ``m.x`` is equivalent to ``m.__dict__["x"]``. A module object does not contain the code object used to initialize the module (since it isn't needed once the initialization is done). Attribute assignment updates the module's namespace dictionary, e.g., ``m.x = 1`` is equivalent to ``m.__dict__["x"] = 1``. .. index:: single: __name__ (module attribute) single: __doc__ (module attribute) single: __file__ (module attribute) single: __annotations__ (module attribute) pair: module; namespace Predefined (writable) attributes: :attr:`__name__` is the module's name; :attr:`__doc__` is the module's documentation string, or ``None`` if unavailable; :attr:`__annotations__` (optional) is a dictionary containing :term:`variable annotations ` collected during module body execution; :attr:`__file__` is the pathname of the file from which the module was loaded, if it was loaded from a file. The :attr:`__file__` attribute may be missing for certain types of modules, such as C modules that are statically linked into the interpreter; for extension modules loaded dynamically from a shared library, it is the pathname of the shared library file. .. index:: single: __dict__ (module attribute) Special read-only attribute: :attr:`~object.__dict__` is the module's namespace as a dictionary object. .. impl-detail:: Because of the way CPython clears module dictionaries, the module dictionary will be cleared when the module falls out of scope even if the dictionary still has live references. To avoid this, copy the dictionary or keep the module around while using its dictionary directly. Custom classes Custom class types are typically created by class definitions (see section :ref:`class`). A class has a namespace implemented by a dictionary object. Class attribute references are translated to lookups in this dictionary, e.g., ``C.x`` is translated to ``C.__dict__["x"]`` (although there are a number of hooks which allow for other means of locating attributes). When the attribute name is not found there, the attribute search continues in the base classes. This search of the base classes uses the C3 method resolution order which behaves correctly even in the presence of 'diamond' inheritance structures where there are multiple inheritance paths leading back to a common ancestor. Additional details on the C3 MRO used by Python can be found in the documentation accompanying the 2.3 release at https://www.python.org/download/releases/2.3/mro/. .. XXX: Could we add that MRO doc as an appendix to the language ref? .. index:: object: class object: class instance object: instance pair: class object; call single: container object: dictionary pair: class; attribute When a class attribute reference (for class :class:`C`, say) would yield a class method object, it is transformed into an instance method object whose :attr:`__self__` attribute is :class:`C`. When it would yield a static method object, it is transformed into the object wrapped by the static method object. See section :ref:`descriptors` for another way in which attributes retrieved from a class may differ from those actually contained in its :attr:`~object.__dict__`. .. index:: triple: class; attribute; assignment Class attribute assignments update the class's dictionary, never the dictionary of a base class. .. index:: pair: class object; call A class object can be called (see above) to yield a class instance (see below). .. index:: single: __name__ (class attribute) single: __module__ (class attribute) single: __dict__ (class attribute) single: __bases__ (class attribute) single: __doc__ (class attribute) single: __annotations__ (class attribute) Special attributes: :attr:`~definition.__name__` is the class name; :attr:`__module__` is the module name in which the class was defined; :attr:`~object.__dict__` is the dictionary containing the class's namespace; :attr:`~class.__bases__` is a tuple containing the base classes, in the order of their occurrence in the base class list; :attr:`__doc__` is the class's documentation string, or ``None`` if undefined; :attr:`__annotations__` (optional) is a dictionary containing :term:`variable annotations ` collected during class body execution. Class instances .. index:: object: class instance object: instance pair: class; instance pair: class instance; attribute A class instance is created by calling a class object (see above). A class instance has a namespace implemented as a dictionary which is the first place in which attribute references are searched. When an attribute is not found there, and the instance's class has an attribute by that name, the search continues with the class attributes. If a class attribute is found that is a user-defined function object, it is transformed into an instance method object whose :attr:`__self__` attribute is the instance. Static method and class method objects are also transformed; see above under "Classes". See section :ref:`descriptors` for another way in which attributes of a class retrieved via its instances may differ from the objects actually stored in the class's :attr:`~object.__dict__`. If no class attribute is found, and the object's class has a :meth:`__getattr__` method, that is called to satisfy the lookup. .. index:: triple: class instance; attribute; assignment Attribute assignments and deletions update the instance's dictionary, never a class's dictionary. If the class has a :meth:`__setattr__` or :meth:`__delattr__` method, this is called instead of updating the instance dictionary directly. .. index:: object: numeric object: sequence object: mapping Class instances can pretend to be numbers, sequences, or mappings if they have methods with certain special names. See section :ref:`specialnames`. .. index:: single: __dict__ (instance attribute) single: __class__ (instance attribute) Special attributes: :attr:`~object.__dict__` is the attribute dictionary; :attr:`~instance.__class__` is the instance's class. I/O objects (also known as file objects) .. index:: builtin: open module: io single: popen() (in module os) single: makefile() (socket method) single: sys.stdin single: sys.stdout single: sys.stderr single: stdio single: stdin (in module sys) single: stdout (in module sys) single: stderr (in module sys) A :term:`file object` represents an open file. Various shortcuts are available to create file objects: the :func:`open` built-in function, and also :func:`os.popen`, :func:`os.fdopen`, and the :meth:`~socket.socket.makefile` method of socket objects (and perhaps by other functions or methods provided by extension modules). The objects ``sys.stdin``, ``sys.stdout`` and ``sys.stderr`` are initialized to file objects corresponding to the interpreter's standard input, output and error streams; they are all open in text mode and therefore follow the interface defined by the :class:`io.TextIOBase` abstract class. Internal types .. index:: single: internal type single: types, internal A few types used internally by the interpreter are exposed to the user. Their definitions may change with future versions of the interpreter, but they are mentioned here for completeness. .. index:: bytecode, object; code, code object Code objects Code objects represent *byte-compiled* executable Python code, or :term:`bytecode`. The difference between a code object and a function object is that the function object contains an explicit reference to the function's globals (the module in which it was defined), while a code object contains no context; also the default argument values are stored in the function object, not in the code object (because they represent values calculated at run-time). Unlike function objects, code objects are immutable and contain no references (directly or indirectly) to mutable objects. .. index:: single: co_argcount (code object attribute) single: co_code (code object attribute) single: co_consts (code object attribute) single: co_filename (code object attribute) single: co_firstlineno (code object attribute) single: co_flags (code object attribute) single: co_lnotab (code object attribute) single: co_name (code object attribute) single: co_names (code object attribute) single: co_nlocals (code object attribute) single: co_stacksize (code object attribute) single: co_varnames (code object attribute) single: co_cellvars (code object attribute) single: co_freevars (code object attribute) Special read-only attributes: :attr:`co_name` gives the function name; :attr:`co_argcount` is the number of positional arguments (including arguments with default values); :attr:`co_nlocals` is the number of local variables used by the function (including arguments); :attr:`co_varnames` is a tuple containing the names of the local variables (starting with the argument names); :attr:`co_cellvars` is a tuple containing the names of local variables that are referenced by nested functions; :attr:`co_freevars` is a tuple containing the names of free variables; :attr:`co_code` is a string representing the sequence of bytecode instructions; :attr:`co_consts` is a tuple containing the literals used by the bytecode; :attr:`co_names` is a tuple containing the names used by the bytecode; :attr:`co_filename` is the filename from which the code was compiled; :attr:`co_firstlineno` is the first line number of the function; :attr:`co_lnotab` is a string encoding the mapping from bytecode offsets to line numbers (for details see the source code of the interpreter); :attr:`co_stacksize` is the required stack size (including local variables); :attr:`co_flags` is an integer encoding a number of flags for the interpreter. .. index:: object: generator The following flag bits are defined for :attr:`co_flags`: bit ``0x04`` is set if the function uses the ``*arguments`` syntax to accept an arbitrary number of positional arguments; bit ``0x08`` is set if the function uses the ``**keywords`` syntax to accept arbitrary keyword arguments; bit ``0x20`` is set if the function is a generator. Future feature declarations (``from __future__ import division``) also use bits in :attr:`co_flags` to indicate whether a code object was compiled with a particular feature enabled: bit ``0x2000`` is set if the function was compiled with future division enabled; bits ``0x10`` and ``0x1000`` were used in earlier versions of Python. Other bits in :attr:`co_flags` are reserved for internal use. .. index:: single: documentation string If a code object represents a function, the first item in :attr:`co_consts` is the documentation string of the function, or ``None`` if undefined. .. _frame-objects: Frame objects .. index:: object: frame Frame objects represent execution frames. They may occur in traceback objects (see below). .. index:: single: f_back (frame attribute) single: f_code (frame attribute) single: f_globals (frame attribute) single: f_locals (frame attribute) single: f_lasti (frame attribute) single: f_builtins (frame attribute) Special read-only attributes: :attr:`f_back` is to the previous stack frame (towards the caller), or ``None`` if this is the bottom stack frame; :attr:`f_code` is the code object being executed in this frame; :attr:`f_locals` is the dictionary used to look up local variables; :attr:`f_globals` is used for global variables; :attr:`f_builtins` is used for built-in (intrinsic) names; :attr:`f_lasti` gives the precise instruction (this is an index into the bytecode string of the code object). .. index:: single: f_trace (frame attribute) single: f_lineno (frame attribute) Special writable attributes: :attr:`f_trace`, if not ``None``, is a function called at the start of each source code line (this is used by the debugger); :attr:`f_lineno` is the current line number of the frame --- writing to this from within a trace function jumps to the given line (only for the bottom-most frame). A debugger can implement a Jump command (aka Set Next Statement) by writing to f_lineno. Frame objects support one method: .. method:: frame.clear() This method clears all references to local variables held by the frame. Also, if the frame belonged to a generator, the generator is finalized. This helps break reference cycles involving frame objects (for example when catching an exception and storing its traceback for later use). :exc:`RuntimeError` is raised if the frame is currently executing. .. versionadded:: 3.4 Traceback objects .. index:: object: traceback pair: stack; trace pair: exception; handler pair: execution; stack single: exc_info (in module sys) single: last_traceback (in module sys) single: sys.exc_info single: sys.last_traceback Traceback objects represent a stack trace of an exception. A traceback object is created when an exception occurs. When the search for an exception handler unwinds the execution stack, at each unwound level a traceback object is inserted in front of the current traceback. When an exception handler is entered, the stack trace is made available to the program. (See section :ref:`try`.) It is accessible as the third item of the tuple returned by ``sys.exc_info()``. When the program contains no suitable handler, the stack trace is written (nicely formatted) to the standard error stream; if the interpreter is interactive, it is also made available to the user as ``sys.last_traceback``. .. index:: single: tb_next (traceback attribute) single: tb_frame (traceback attribute) single: tb_lineno (traceback attribute) single: tb_lasti (traceback attribute) statement: try Special read-only attributes: :attr:`tb_next` is the next level in the stack trace (towards the frame where the exception occurred), or ``None`` if there is no next level; :attr:`tb_frame` points to the execution frame of the current level; :attr:`tb_lineno` gives the line number where the exception occurred; :attr:`tb_lasti` indicates the precise instruction. The line number and last instruction in the traceback may differ from the line number of its frame object if the exception occurred in a :keyword:`try` statement with no matching except clause or with a finally clause. Slice objects .. index:: builtin: slice Slice objects are used to represent slices for :meth:`__getitem__` methods. They are also created by the built-in :func:`slice` function. .. index:: single: start (slice object attribute) single: stop (slice object attribute) single: step (slice object attribute) Special read-only attributes: :attr:`~slice.start` is the lower bound; :attr:`~slice.stop` is the upper bound; :attr:`~slice.step` is the step value; each is ``None`` if omitted. These attributes can have any type. Slice objects support one method: .. method:: slice.indices(self, length) This method takes a single integer argument *length* and computes information about the slice that the slice object would describe if applied to a sequence of *length* items. It returns a tuple of three integers; respectively these are the *start* and *stop* indices and the *step* or stride length of the slice. Missing or out-of-bounds indices are handled in a manner consistent with regular slices. Static method objects Static method objects provide a way of defeating the transformation of function objects to method objects described above. A static method object is a wrapper around any other object, usually a user-defined method object. When a static method object is retrieved from a class or a class instance, the object actually returned is the wrapped object, which is not subject to any further transformation. Static method objects are not themselves callable, although the objects they wrap usually are. Static method objects are created by the built-in :func:`staticmethod` constructor. Class method objects A class method object, like a static method object, is a wrapper around another object that alters the way in which that object is retrieved from classes and class instances. The behaviour of class method objects upon such retrieval is described above, under "User-defined methods". Class method objects are created by the built-in :func:`classmethod` constructor. .. _specialnames: Special method names ==================== .. index:: pair: operator; overloading single: __getitem__() (mapping object method) A class can implement certain operations that are invoked by special syntax (such as arithmetic operations or subscripting and slicing) by defining methods with special names. This is Python's approach to :dfn:`operator overloading`, allowing classes to define their own behavior with respect to language operators. For instance, if a class defines a method named :meth:`__getitem__`, and ``x`` is an instance of this class, then ``x[i]`` is roughly equivalent to ``type(x).__getitem__(x, i)``. Except where mentioned, attempts to execute an operation raise an exception when no appropriate method is defined (typically :exc:`AttributeError` or :exc:`TypeError`). Setting a special method to ``None`` indicates that the corresponding operation is not available. For example, if a class sets :meth:`__iter__` to ``None``, the class is not iterable, so calling :func:`iter` on its instances will raise a :exc:`TypeError` (without falling back to :meth:`__getitem__`). [#]_ When implementing a class that emulates any built-in type, it is important that the emulation only be implemented to the degree that it makes sense for the object being modelled. For example, some sequences may work well with retrieval of individual elements, but extracting a slice may not make sense. (One example of this is the :class:`~xml.dom.NodeList` interface in the W3C's Document Object Model.) .. _customization: Basic customization ------------------- .. method:: object.__new__(cls[, ...]) .. index:: pair: subclassing; immutable types Called to create a new instance of class *cls*. :meth:`__new__` is a static method (special-cased so you need not declare it as such) that takes the class of which an instance was requested as its first argument. The remaining arguments are those passed to the object constructor expression (the call to the class). The return value of :meth:`__new__` should be the new object instance (usually an instance of *cls*). Typical implementations create a new instance of the class by invoking the superclass's :meth:`__new__` method using ``super().__new__(cls[, ...])`` with appropriate arguments and then modifying the newly-created instance as necessary before returning it. If :meth:`__new__` returns an instance of *cls*, then the new instance's :meth:`__init__` method will be invoked like ``__init__(self[, ...])``, where *self* is the new instance and the remaining arguments are the same as were passed to :meth:`__new__`. If :meth:`__new__` does not return an instance of *cls*, then the new instance's :meth:`__init__` method will not be invoked. :meth:`__new__` is intended mainly to allow subclasses of immutable types (like int, str, or tuple) to customize instance creation. It is also commonly overridden in custom metaclasses in order to customize class creation. .. method:: object.__init__(self[, ...]) .. index:: pair: class; constructor Called after the instance has been created (by :meth:`__new__`), but before it is returned to the caller. The arguments are those passed to the class constructor expression. If a base class has an :meth:`__init__` method, the derived class's :meth:`__init__` method, if any, must explicitly call it to ensure proper initialization of the base class part of the instance; for example: ``super().__init__([args...])``. Because :meth:`__new__` and :meth:`__init__` work together in constructing objects (:meth:`__new__` to create it, and :meth:`__init__` to customize it), no non-``None`` value may be returned by :meth:`__init__`; doing so will cause a :exc:`TypeError` to be raised at runtime. .. method:: object.__del__(self) .. index:: single: destructor single: finalizer statement: del Called when the instance is about to be destroyed. This is also called a finalizer or (improperly) a destructor. If a base class has a :meth:`__del__` method, the derived class's :meth:`__del__` method, if any, must explicitly call it to ensure proper deletion of the base class part of the instance. It is possible (though not recommended!) for the :meth:`__del__` method to postpone destruction of the instance by creating a new reference to it. This is called object *resurrection*. It is implementation-dependent whether :meth:`__del__` is called a second time when a resurrected object is about to be destroyed; the current :term:`CPython` implementation only calls it once. It is not guaranteed that :meth:`__del__` methods are called for objects that still exist when the interpreter exits. .. note:: ``del x`` doesn't directly call ``x.__del__()`` --- the former decrements the reference count for ``x`` by one, and the latter is only called when ``x``'s reference count reaches zero. .. impl-detail:: It is possible for a reference cycle to prevent the reference count of an object from going to zero. In this case, the cycle will be later detected and deleted by the :term:`cyclic garbage collector `. A common cause of reference cycles is when an exception has been caught in a local variable. The frame's locals then reference the exception, which references its own traceback, which references the locals of all frames caught in the traceback. .. seealso:: Documentation for the :mod:`gc` module. .. warning:: Due to the precarious circumstances under which :meth:`__del__` methods are invoked, exceptions that occur during their execution are ignored, and a warning is printed to ``sys.stderr`` instead. In particular: * :meth:`__del__` can be invoked when arbitrary code is being executed, including from any arbitrary thread. If :meth:`__del__` needs to take a lock or invoke any other blocking resource, it may deadlock as the resource may already be taken by the code that gets interrupted to execute :meth:`__del__`. * :meth:`__del__` can be executed during interpreter shutdown. As a consequence, the global variables it needs to access (including other modules) may already have been deleted or set to ``None``. Python guarantees that globals whose name begins with a single underscore are deleted from their module before other globals are deleted; if no other references to such globals exist, this may help in assuring that imported modules are still available at the time when the :meth:`__del__` method is called. .. index:: single: repr() (built-in function); __repr__() (object method) .. method:: object.__repr__(self) Called by the :func:`repr` built-in function to compute the "official" string representation of an object. If at all possible, this should look like a valid Python expression that could be used to recreate an object with the same value (given an appropriate environment). If this is not possible, a string of the form ``<...some useful description...>`` should be returned. The return value must be a string object. If a class defines :meth:`__repr__` but not :meth:`__str__`, then :meth:`__repr__` is also used when an "informal" string representation of instances of that class is required. This is typically used for debugging, so it is important that the representation is information-rich and unambiguous. .. index:: single: string; __str__() (object method) single: format() (built-in function); __str__() (object method) single: print() (built-in function); __str__() (object method) .. method:: object.__str__(self) Called by :func:`str(object) ` and the built-in functions :func:`format` and :func:`print` to compute the "informal" or nicely printable string representation of an object. The return value must be a :ref:`string ` object. This method differs from :meth:`object.__repr__` in that there is no expectation that :meth:`__str__` return a valid Python expression: a more convenient or concise representation can be used. The default implementation defined by the built-in type :class:`object` calls :meth:`object.__repr__`. .. XXX what about subclasses of string? .. method:: object.__bytes__(self) .. index:: builtin: bytes Called by :ref:`bytes ` to compute a byte-string representation of an object. This should return a :class:`bytes` object. .. index:: single: string; __format__() (object method) pair: string; conversion builtin: print .. method:: object.__format__(self, format_spec) Called by the :func:`format` built-in function, and by extension, evaluation of :ref:`formatted string literals ` and the :meth:`str.format` method, to produce a "formatted" string representation of an object. The ``format_spec`` argument is a string that contains a description of the formatting options desired. The interpretation of the ``format_spec`` argument is up to the type implementing :meth:`__format__`, however most classes will either delegate formatting to one of the built-in types, or use a similar formatting option syntax. See :ref:`formatspec` for a description of the standard formatting syntax. The return value must be a string object. .. versionchanged:: 3.4 The __format__ method of ``object`` itself raises a :exc:`TypeError` if passed any non-empty string. .. _richcmpfuncs: .. method:: object.__lt__(self, other) object.__le__(self, other) object.__eq__(self, other) object.__ne__(self, other) object.__gt__(self, other) object.__ge__(self, other) .. index:: single: comparisons These are the so-called "rich comparison" methods. The correspondence between operator symbols and method names is as follows: ``xy`` calls ``x.__gt__(y)``, and ``x>=y`` calls ``x.__ge__(y)``. A rich comparison method may return the singleton ``NotImplemented`` if it does not implement the operation for a given pair of arguments. By convention, ``False`` and ``True`` are returned for a successful comparison. However, these methods can return any value, so if the comparison operator is used in a Boolean context (e.g., in the condition of an ``if`` statement), Python will call :func:`bool` on the value to determine if the result is true or false. By default, :meth:`__ne__` delegates to :meth:`__eq__` and inverts the result unless it is ``NotImplemented``. There are no other implied relationships among the comparison operators, for example, the truth of ``(x.__hash__``. If a class that does not override :meth:`__eq__` wishes to suppress hash support, it should include ``__hash__ = None`` in the class definition. A class which defines its own :meth:`__hash__` that explicitly raises a :exc:`TypeError` would be incorrectly identified as hashable by an ``isinstance(obj, collections.Hashable)`` call. .. note:: By default, the :meth:`__hash__` values of str, bytes and datetime objects are "salted" with an unpredictable random value. Although they remain constant within an individual Python process, they are not predictable between repeated invocations of Python. This is intended to provide protection against a denial-of-service caused by carefully-chosen inputs that exploit the worst case performance of a dict insertion, O(n^2) complexity. See http://www.ocert.org/advisories/ocert-2011-003.html for details. Changing hash values affects the iteration order of dicts, sets and other mappings. Python has never made guarantees about this ordering (and it typically varies between 32-bit and 64-bit builds). See also :envvar:`PYTHONHASHSEED`. .. versionchanged:: 3.3 Hash randomization is enabled by default. .. method:: object.__bool__(self) .. index:: single: __len__() (mapping object method) Called to implement truth value testing and the built-in operation ``bool()``; should return ``False`` or ``True``. When this method is not defined, :meth:`__len__` is called, if it is defined, and the object is considered true if its result is nonzero. If a class defines neither :meth:`__len__` nor :meth:`__bool__`, all its instances are considered true. .. _attribute-access: Customizing attribute access ---------------------------- The following methods can be defined to customize the meaning of attribute access (use of, assignment to, or deletion of ``x.name``) for class instances. .. XXX explain how descriptors interfere here! .. method:: object.__getattr__(self, name) Called when the default attribute access fails with an :exc:`AttributeError` (either :meth:`__getattribute__` raises an :exc:`AttributeError` because *name* is not an instance attribute or an attribute in the class tree for ``self``; or :meth:`__get__` of a *name* property raises :exc:`AttributeError`). This method should either return the (computed) attribute value or raise an :exc:`AttributeError` exception. Note that if the attribute is found through the normal mechanism, :meth:`__getattr__` is not called. (This is an intentional asymmetry between :meth:`__getattr__` and :meth:`__setattr__`.) This is done both for efficiency reasons and because otherwise :meth:`__getattr__` would have no way to access other attributes of the instance. Note that at least for instance variables, you can fake total control by not inserting any values in the instance attribute dictionary (but instead inserting them in another object). See the :meth:`__getattribute__` method below for a way to actually get total control over attribute access. .. method:: object.__getattribute__(self, name) Called unconditionally to implement attribute accesses for instances of the class. If the class also defines :meth:`__getattr__`, the latter will not be called unless :meth:`__getattribute__` either calls it explicitly or raises an :exc:`AttributeError`. This method should return the (computed) attribute value or raise an :exc:`AttributeError` exception. In order to avoid infinite recursion in this method, its implementation should always call the base class method with the same name to access any attributes it needs, for example, ``object.__getattribute__(self, name)``. .. note:: This method may still be bypassed when looking up special methods as the result of implicit invocation via language syntax or built-in functions. See :ref:`special-lookup`. .. method:: object.__setattr__(self, name, value) Called when an attribute assignment is attempted. This is called instead of the normal mechanism (i.e. store the value in the instance dictionary). *name* is the attribute name, *value* is the value to be assigned to it. If :meth:`__setattr__` wants to assign to an instance attribute, it should call the base class method with the same name, for example, ``object.__setattr__(self, name, value)``. .. method:: object.__delattr__(self, name) Like :meth:`__setattr__` but for attribute deletion instead of assignment. This should only be implemented if ``del obj.name`` is meaningful for the object. .. method:: object.__dir__(self) Called when :func:`dir` is called on the object. A sequence must be returned. :func:`dir` converts the returned sequence to a list and sorts it. Customizing module attribute access ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. index:: single: __class__ (module attribute) For a more fine grained customization of the module behavior (setting attributes, properties, etc.), one can set the ``__class__`` attribute of a module object to a subclass of :class:`types.ModuleType`. For example:: import sys from types import ModuleType class VerboseModule(ModuleType): def __repr__(self): return f'Verbose {self.__name__}' def __setattr__(self, attr, value): print(f'Setting {attr}...') setattr(self, attr, value) sys.modules[__name__].__class__ = VerboseModule .. note:: Setting module ``__class__`` only affects lookups made using the attribute access syntax -- directly accessing the module globals (whether by code within the module, or via a reference to the module's globals dictionary) is unaffected. .. versionchanged:: 3.5 ``__class__`` module attribute is now writable. .. _descriptors: Implementing Descriptors ^^^^^^^^^^^^^^^^^^^^^^^^ The following methods only apply when an instance of the class containing the method (a so-called *descriptor* class) appears in an *owner* class (the descriptor must be in either the owner's class dictionary or in the class dictionary for one of its parents). In the examples below, "the attribute" refers to the attribute whose name is the key of the property in the owner class' :attr:`~object.__dict__`. .. method:: object.__get__(self, instance, owner) Called to get the attribute of the owner class (class attribute access) or of an instance of that class (instance attribute access). *owner* is always the owner class, while *instance* is the instance that the attribute was accessed through, or ``None`` when the attribute is accessed through the *owner*. This method should return the (computed) attribute value or raise an :exc:`AttributeError` exception. .. method:: object.__set__(self, instance, value) Called to set the attribute on an instance *instance* of the owner class to a new value, *value*. .. method:: object.__delete__(self, instance) Called to delete the attribute on an instance *instance* of the owner class. .. method:: object.__set_name__(self, owner, name) Called at the time the owning class *owner* is created. The descriptor has been assigned to *name*. .. versionadded:: 3.6 The attribute :attr:`__objclass__` is interpreted by the :mod:`inspect` module as specifying the class where this object was defined (setting this appropriately can assist in runtime introspection of dynamic class attributes). For callables, it may indicate that an instance of the given type (or a subclass) is expected or required as the first positional argument (for example, CPython sets this attribute for unbound methods that are implemented in C). .. _descriptor-invocation: Invoking Descriptors ^^^^^^^^^^^^^^^^^^^^ In general, a descriptor is an object attribute with "binding behavior", one whose attribute access has been overridden by methods in the descriptor protocol: :meth:`__get__`, :meth:`__set__`, and :meth:`__delete__`. If any of those methods are defined for an object, it is said to be a descriptor. The default behavior for attribute access is to get, set, or delete the attribute from an object's dictionary. For instance, ``a.x`` has a lookup chain starting with ``a.__dict__['x']``, then ``type(a).__dict__['x']``, and continuing through the base classes of ``type(a)`` excluding metaclasses. However, if the looked-up value is an object defining one of the descriptor methods, then Python may override the default behavior and invoke the descriptor method instead. Where this occurs in the precedence chain depends on which descriptor methods were defined and how they were called. The starting point for descriptor invocation is a binding, ``a.x``. How the arguments are assembled depends on ``a``: Direct Call The simplest and least common call is when user code directly invokes a descriptor method: ``x.__get__(a)``. Instance Binding If binding to an object instance, ``a.x`` is transformed into the call: ``type(a).__dict__['x'].__get__(a, type(a))``. Class Binding If binding to a class, ``A.x`` is transformed into the call: ``A.__dict__['x'].__get__(None, A)``. Super Binding If ``a`` is an instance of :class:`super`, then the binding ``super(B, obj).m()`` searches ``obj.__class__.__mro__`` for the base class ``A`` immediately preceding ``B`` and then invokes the descriptor with the call: ``A.__dict__['m'].__get__(obj, obj.__class__)``. For instance bindings, the precedence of descriptor invocation depends on the which descriptor methods are defined. A descriptor can define any combination of :meth:`__get__`, :meth:`__set__` and :meth:`__delete__`. If it does not define :meth:`__get__`, then accessing the attribute will return the descriptor object itself unless there is a value in the object's instance dictionary. If the descriptor defines :meth:`__set__` and/or :meth:`__delete__`, it is a data descriptor; if it defines neither, it is a non-data descriptor. Normally, data descriptors define both :meth:`__get__` and :meth:`__set__`, while non-data descriptors have just the :meth:`__get__` method. Data descriptors with :meth:`__set__` and :meth:`__get__` defined always override a redefinition in an instance dictionary. In contrast, non-data descriptors can be overridden by instances. Python methods (including :func:`staticmethod` and :func:`classmethod`) are implemented as non-data descriptors. Accordingly, instances can redefine and override methods. This allows individual instances to acquire behaviors that differ from other instances of the same class. The :func:`property` function is implemented as a data descriptor. Accordingly, instances cannot override the behavior of a property. .. _slots: __slots__ ^^^^^^^^^ *__slots__* allow us to explicitly declare data members (like properties) and deny the creation of *__dict__* and *__weakref__* (unless explicitly declared in *__slots__* or available in a parent.) The space saved over using *__dict__* can be significant. .. data:: object.__slots__ This class variable can be assigned a string, iterable, or sequence of strings with variable names used by instances. *__slots__* reserves space for the declared variables and prevents the automatic creation of *__dict__* and *__weakref__* for each instance. Notes on using *__slots__* """""""""""""""""""""""""" * When inheriting from a class without *__slots__*, the *__dict__* and *__weakref__* attribute of the instances will always be accessible. * Without a *__dict__* variable, instances cannot be assigned new variables not listed in the *__slots__* definition. Attempts to assign to an unlisted variable name raises :exc:`AttributeError`. If dynamic assignment of new variables is desired, then add ``'__dict__'`` to the sequence of strings in the *__slots__* declaration. * Without a *__weakref__* variable for each instance, classes defining *__slots__* do not support weak references to its instances. If weak reference support is needed, then add ``'__weakref__'`` to the sequence of strings in the *__slots__* declaration. * *__slots__* are implemented at the class level by creating descriptors (:ref:`descriptors`) for each variable name. As a result, class attributes cannot be used to set default values for instance variables defined by *__slots__*; otherwise, the class attribute would overwrite the descriptor assignment. * The action of a *__slots__* declaration is not limited to the class where it is defined. *__slots__* declared in parents are available in child classes. However, child subclasses will get a *__dict__* and *__weakref__* unless they also define *__slots__* (which should only contain names of any *additional* slots). * If a class defines a slot also defined in a base class, the instance variable defined by the base class slot is inaccessible (except by retrieving its descriptor directly from the base class). This renders the meaning of the program undefined. In the future, a check may be added to prevent this. * Nonempty *__slots__* does not work for classes derived from "variable-length" built-in types such as :class:`int`, :class:`bytes` and :class:`tuple`. * Any non-string iterable may be assigned to *__slots__*. Mappings may also be used; however, in the future, special meaning may be assigned to the values corresponding to each key. * *__class__* assignment works only if both classes have the same *__slots__*. * Multiple inheritance with multiple slotted parent classes can be used, but only one parent is allowed to have attributes created by slots (the other bases must have empty slot layouts) - violations raise :exc:`TypeError`. .. _class-customization: Customizing class creation -------------------------- Whenever a class inherits from another class, *__init_subclass__* is called on that class. This way, it is possible to write classes which change the behavior of subclasses. This is closely related to class decorators, but where class decorators only affect the specific class they're applied to, ``__init_subclass__`` solely applies to future subclasses of the class defining the method. .. classmethod:: object.__init_subclass__(cls) This method is called whenever the containing class is subclassed. *cls* is then the new subclass. If defined as a normal instance method, this method is implicitly converted to a class method. Keyword arguments which are given to a new class are passed to the parent's class ``__init_subclass__``. For compatibility with other classes using ``__init_subclass__``, one should take out the needed keyword arguments and pass the others over to the base class, as in:: class Philosopher: def __init_subclass__(cls, default_name, **kwargs): super().__init_subclass__(**kwargs) cls.default_name = default_name class AustralianPhilosopher(Philosopher, default_name="Bruce"): pass The default implementation ``object.__init_subclass__`` does nothing, but raises an error if it is called with any arguments. .. note:: The metaclass hint ``metaclass`` is consumed by the rest of the type machinery, and is never passed to ``__init_subclass__`` implementations. The actual metaclass (rather than the explicit hint) can be accessed as ``type(cls)``. .. versionadded:: 3.6 .. _metaclasses: Metaclasses ^^^^^^^^^^^ .. index:: single: metaclass builtin: type By default, classes are constructed using :func:`type`. The class body is executed in a new namespace and the class name is bound locally to the result of ``type(name, bases, namespace)``. The class creation process can be customized by passing the ``metaclass`` keyword argument in the class definition line, or by inheriting from an existing class that included such an argument. In the following example, both ``MyClass`` and ``MySubclass`` are instances of ``Meta``:: class Meta(type): pass class MyClass(metaclass=Meta): pass class MySubclass(MyClass): pass Any other keyword arguments that are specified in the class definition are passed through to all metaclass operations described below. When a class definition is executed, the following steps occur: * the appropriate metaclass is determined * the class namespace is prepared * the class body is executed * the class object is created Determining the appropriate metaclass ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. index:: single: metaclass hint The appropriate metaclass for a class definition is determined as follows: * if no bases and no explicit metaclass are given, then :func:`type` is used * if an explicit metaclass is given and it is *not* an instance of :func:`type`, then it is used directly as the metaclass * if an instance of :func:`type` is given as the explicit metaclass, or bases are defined, then the most derived metaclass is used The most derived metaclass is selected from the explicitly specified metaclass (if any) and the metaclasses (i.e. ``type(cls)``) of all specified base classes. The most derived metaclass is one which is a subtype of *all* of these candidate metaclasses. If none of the candidate metaclasses meets that criterion, then the class definition will fail with ``TypeError``. .. _prepare: Preparing the class namespace ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. index:: single: __prepare__ (metaclass method) Once the appropriate metaclass has been identified, then the class namespace is prepared. If the metaclass has a ``__prepare__`` attribute, it is called as ``namespace = metaclass.__prepare__(name, bases, **kwds)`` (where the additional keyword arguments, if any, come from the class definition). If the metaclass has no ``__prepare__`` attribute, then the class namespace is initialised as an empty ordered mapping. .. seealso:: :pep:`3115` - Metaclasses in Python 3000 Introduced the ``__prepare__`` namespace hook Executing the class body ^^^^^^^^^^^^^^^^^^^^^^^^ .. index:: single: class; body The class body is executed (approximately) as ``exec(body, globals(), namespace)``. The key difference from a normal call to :func:`exec` is that lexical scoping allows the class body (including any methods) to reference names from the current and outer scopes when the class definition occurs inside a function. However, even when the class definition occurs inside the function, methods defined inside the class still cannot see names defined at the class scope. Class variables must be accessed through the first parameter of instance or class methods, or through the implicit lexically scoped ``__class__`` reference described in the next section. .. _class-object-creation: Creating the class object ^^^^^^^^^^^^^^^^^^^^^^^^^ .. index:: single: __class__ (method cell) single: __classcell__ (class namespace entry) Once the class namespace has been populated by executing the class body, the class object is created by calling ``metaclass(name, bases, namespace, **kwds)`` (the additional keywords passed here are the same as those passed to ``__prepare__``). This class object is the one that will be referenced by the zero-argument form of :func:`super`. ``__class__`` is an implicit closure reference created by the compiler if any methods in a class body refer to either ``__class__`` or ``super``. This allows the zero argument form of :func:`super` to correctly identify the class being defined based on lexical scoping, while the class or instance that was used to make the current call is identified based on the first argument passed to the method. .. impl-detail:: In CPython 3.6 and later, the ``__class__`` cell is passed to the metaclass as a ``__classcell__`` entry in the class namespace. If present, this must be propagated up to the ``type.__new__`` call in order for the class to be initialised correctly. Failing to do so will result in a :exc:`DeprecationWarning` in Python 3.6, and a :exc:`RuntimeError` in Python 3.8. When using the default metaclass :class:`type`, or any metaclass that ultimately calls ``type.__new__``, the following additional customisation steps are invoked after creating the class object: * first, ``type.__new__`` collects all of the descriptors in the class namespace that define a :meth:`~object.__set_name__` method; * second, all of these ``__set_name__`` methods are called with the class being defined and the assigned name of that particular descriptor; and * finally, the :meth:`~object.__init_subclass__` hook is called on the immediate parent of the new class in its method resolution order. After the class object is created, it is passed to the class decorators included in the class definition (if any) and the resulting object is bound in the local namespace as the defined class. When a new class is created by ``type.__new__``, the object provided as the namespace parameter is copied to a new ordered mapping and the original object is discarded. The new copy is wrapped in a read-only proxy, which becomes the :attr:`~object.__dict__` attribute of the class object. .. seealso:: :pep:`3135` - New super Describes the implicit ``__class__`` closure reference Metaclass example ^^^^^^^^^^^^^^^^^ The potential uses for metaclasses are boundless. Some ideas that have been explored include enum, logging, interface checking, automatic delegation, automatic property creation, proxies, frameworks, and automatic resource locking/synchronization. Here is an example of a metaclass that uses an :class:`collections.OrderedDict` to remember the order that class variables are defined:: class OrderedClass(type): @classmethod def __prepare__(metacls, name, bases, **kwds): return collections.OrderedDict() def __new__(cls, name, bases, namespace, **kwds): result = type.__new__(cls, name, bases, dict(namespace)) result.members = tuple(namespace) return result class A(metaclass=OrderedClass): def one(self): pass def two(self): pass def three(self): pass def four(self): pass >>> A.members ('__module__', 'one', 'two', 'three', 'four') When the class definition for *A* gets executed, the process begins with calling the metaclass's :meth:`__prepare__` method which returns an empty :class:`collections.OrderedDict`. That mapping records the methods and attributes of *A* as they are defined within the body of the class statement. Once those definitions are executed, the ordered dictionary is fully populated and the metaclass's :meth:`__new__` method gets invoked. That method builds the new type and it saves the ordered dictionary keys in an attribute called ``members``. Customizing instance and subclass checks ---------------------------------------- The following methods are used to override the default behavior of the :func:`isinstance` and :func:`issubclass` built-in functions. In particular, the metaclass :class:`abc.ABCMeta` implements these methods in order to allow the addition of Abstract Base Classes (ABCs) as "virtual base classes" to any class or type (including built-in types), including other ABCs. .. method:: class.__instancecheck__(self, instance) Return true if *instance* should be considered a (direct or indirect) instance of *class*. If defined, called to implement ``isinstance(instance, class)``. .. method:: class.__subclasscheck__(self, subclass) Return true if *subclass* should be considered a (direct or indirect) subclass of *class*. If defined, called to implement ``issubclass(subclass, class)``. Note that these methods are looked up on the type (metaclass) of a class. They cannot be defined as class methods in the actual class. This is consistent with the lookup of special methods that are called on instances, only in this case the instance is itself a class. .. seealso:: :pep:`3119` - Introducing Abstract Base Classes Includes the specification for customizing :func:`isinstance` and :func:`issubclass` behavior through :meth:`~class.__instancecheck__` and :meth:`~class.__subclasscheck__`, with motivation for this functionality in the context of adding Abstract Base Classes (see the :mod:`abc` module) to the language. .. _callable-types: Emulating callable objects -------------------------- .. method:: object.__call__(self[, args...]) .. index:: pair: call; instance Called when the instance is "called" as a function; if this method is defined, ``x(arg1, arg2, ...)`` is a shorthand for ``x.__call__(arg1, arg2, ...)``. .. _sequence-types: Emulating container types ------------------------- The following methods can be defined to implement container objects. Containers usually are sequences (such as lists or tuples) or mappings (like dictionaries), but can represent other containers as well. The first set of methods is used either to emulate a sequence or to emulate a mapping; the difference is that for a sequence, the allowable keys should be the integers *k* for which ``0 <= k < N`` where *N* is the length of the sequence, or slice objects, which define a range of items. It is also recommended that mappings provide the methods :meth:`keys`, :meth:`values`, :meth:`items`, :meth:`get`, :meth:`clear`, :meth:`setdefault`, :meth:`pop`, :meth:`popitem`, :meth:`!copy`, and :meth:`update` behaving similar to those for Python's standard dictionary objects. The :mod:`collections` module provides a :class:`~collections.abc.MutableMapping` abstract base class to help create those methods from a base set of :meth:`__getitem__`, :meth:`__setitem__`, :meth:`__delitem__`, and :meth:`keys`. Mutable sequences should provide methods :meth:`append`, :meth:`count`, :meth:`index`, :meth:`extend`, :meth:`insert`, :meth:`pop`, :meth:`remove`, :meth:`reverse` and :meth:`sort`, like Python standard list objects. Finally, sequence types should implement addition (meaning concatenation) and multiplication (meaning repetition) by defining the methods :meth:`__add__`, :meth:`__radd__`, :meth:`__iadd__`, :meth:`__mul__`, :meth:`__rmul__` and :meth:`__imul__` described below; they should not define other numerical operators. It is recommended that both mappings and sequences implement the :meth:`__contains__` method to allow efficient use of the ``in`` operator; for mappings, ``in`` should search the mapping's keys; for sequences, it should search through the values. It is further recommended that both mappings and sequences implement the :meth:`__iter__` method to allow efficient iteration through the container; for mappings, :meth:`__iter__` should be the same as :meth:`keys`; for sequences, it should iterate through the values. .. method:: object.__len__(self) .. index:: builtin: len single: __bool__() (object method) Called to implement the built-in function :func:`len`. Should return the length of the object, an integer ``>=`` 0. Also, an object that doesn't define a :meth:`__bool__` method and whose :meth:`__len__` method returns zero is considered to be false in a Boolean context. .. impl-detail:: In CPython, the length is required to be at most :attr:`sys.maxsize`. If the length is larger than :attr:`!sys.maxsize` some features (such as :func:`len`) may raise :exc:`OverflowError`. To prevent raising :exc:`!OverflowError` by truth value testing, an object must define a :meth:`__bool__` method. .. method:: object.__length_hint__(self) Called to implement :func:`operator.length_hint`. Should return an estimated length for the object (which may be greater or less than the actual length). The length must be an integer ``>=`` 0. This method is purely an optimization and is never required for correctness. .. versionadded:: 3.4 .. note:: Slicing is done exclusively with the following three methods. A call like :: a[1:2] = b is translated to :: a[slice(1, 2, None)] = b and so forth. Missing slice items are always filled in with ``None``. .. method:: object.__getitem__(self, key) .. index:: object: slice Called to implement evaluation of ``self[key]``. For sequence types, the accepted keys should be integers and slice objects. Note that the special interpretation of negative indexes (if the class wishes to emulate a sequence type) is up to the :meth:`__getitem__` method. If *key* is of an inappropriate type, :exc:`TypeError` may be raised; if of a value outside the set of indexes for the sequence (after any special interpretation of negative values), :exc:`IndexError` should be raised. For mapping types, if *key* is missing (not in the container), :exc:`KeyError` should be raised. .. note:: :keyword:`for` loops expect that an :exc:`IndexError` will be raised for illegal indexes to allow proper detection of the end of the sequence. .. method:: object.__missing__(self, key) Called by :class:`dict`\ .\ :meth:`__getitem__` to implement ``self[key]`` for dict subclasses when key is not in the dictionary. .. method:: object.__setitem__(self, key, value) Called to implement assignment to ``self[key]``. Same note as for :meth:`__getitem__`. This should only be implemented for mappings if the objects support changes to the values for keys, or if new keys can be added, or for sequences if elements can be replaced. The same exceptions should be raised for improper *key* values as for the :meth:`__getitem__` method. .. method:: object.__delitem__(self, key) Called to implement deletion of ``self[key]``. Same note as for :meth:`__getitem__`. This should only be implemented for mappings if the objects support removal of keys, or for sequences if elements can be removed from the sequence. The same exceptions should be raised for improper *key* values as for the :meth:`__getitem__` method. .. method:: object.__iter__(self) This method is called when an iterator is required for a container. This method should return a new iterator object that can iterate over all the objects in the container. For mappings, it should iterate over the keys of the container. Iterator objects also need to implement this method; they are required to return themselves. For more information on iterator objects, see :ref:`typeiter`. .. method:: object.__reversed__(self) Called (if present) by the :func:`reversed` built-in to implement reverse iteration. It should return a new iterator object that iterates over all the objects in the container in reverse order. If the :meth:`__reversed__` method is not provided, the :func:`reversed` built-in will fall back to using the sequence protocol (:meth:`__len__` and :meth:`__getitem__`). Objects that support the sequence protocol should only provide :meth:`__reversed__` if they can provide an implementation that is more efficient than the one provided by :func:`reversed`. The membership test operators (:keyword:`in` and :keyword:`not in`) are normally implemented as an iteration through a sequence. However, container objects can supply the following special method with a more efficient implementation, which also does not require the object be a sequence. .. method:: object.__contains__(self, item) Called to implement membership test operators. Should return true if *item* is in *self*, false otherwise. For mapping objects, this should consider the keys of the mapping rather than the values or the key-item pairs. For objects that don't define :meth:`__contains__`, the membership test first tries iteration via :meth:`__iter__`, then the old sequence iteration protocol via :meth:`__getitem__`, see :ref:`this section in the language reference `. .. _numeric-types: Emulating numeric types ----------------------- The following methods can be defined to emulate numeric objects. Methods corresponding to operations that are not supported by the particular kind of number implemented (e.g., bitwise operations for non-integral numbers) should be left undefined. .. method:: object.__add__(self, other) object.__sub__(self, other) object.__mul__(self, other) object.__matmul__(self, other) object.__truediv__(self, other) object.__floordiv__(self, other) object.__mod__(self, other) object.__divmod__(self, other) object.__pow__(self, other[, modulo]) object.__lshift__(self, other) object.__rshift__(self, other) object.__and__(self, other) object.__xor__(self, other) object.__or__(self, other) .. index:: builtin: divmod builtin: pow builtin: pow These methods are called to implement the binary arithmetic operations (``+``, ``-``, ``*``, ``@``, ``/``, ``//``, ``%``, :func:`divmod`, :func:`pow`, ``**``, ``<<``, ``>>``, ``&``, ``^``, ``|``). For instance, to evaluate the expression ``x + y``, where *x* is an instance of a class that has an :meth:`__add__` method, ``x.__add__(y)`` is called. The :meth:`__divmod__` method should be the equivalent to using :meth:`__floordiv__` and :meth:`__mod__`; it should not be related to :meth:`__truediv__`. Note that :meth:`__pow__` should be defined to accept an optional third argument if the ternary version of the built-in :func:`pow` function is to be supported. If one of those methods does not support the operation with the supplied arguments, it should return ``NotImplemented``. .. method:: object.__radd__(self, other) object.__rsub__(self, other) object.__rmul__(self, other) object.__rmatmul__(self, other) object.__rtruediv__(self, other) object.__rfloordiv__(self, other) object.__rmod__(self, other) object.__rdivmod__(self, other) object.__rpow__(self, other) object.__rlshift__(self, other) object.__rrshift__(self, other) object.__rand__(self, other) object.__rxor__(self, other) object.__ror__(self, other) .. index:: builtin: divmod builtin: pow These methods are called to implement the binary arithmetic operations (``+``, ``-``, ``*``, ``@``, ``/``, ``//``, ``%``, :func:`divmod`, :func:`pow`, ``**``, ``<<``, ``>>``, ``&``, ``^``, ``|``) with reflected (swapped) operands. These functions are only called if the left operand does not support the corresponding operation [#]_ and the operands are of different types. [#]_ For instance, to evaluate the expression ``x - y``, where *y* is an instance of a class that has an :meth:`__rsub__` method, ``y.__rsub__(x)`` is called if ``x.__sub__(y)`` returns *NotImplemented*. .. index:: builtin: pow Note that ternary :func:`pow` will not try calling :meth:`__rpow__` (the coercion rules would become too complicated). .. note:: If the right operand's type is a subclass of the left operand's type and that subclass provides the reflected method for the operation, this method will be called before the left operand's non-reflected method. This behavior allows subclasses to override their ancestors' operations. .. method:: object.__iadd__(self, other) object.__isub__(self, other) object.__imul__(self, other) object.__imatmul__(self, other) object.__itruediv__(self, other) object.__ifloordiv__(self, other) object.__imod__(self, other) object.__ipow__(self, other[, modulo]) object.__ilshift__(self, other) object.__irshift__(self, other) object.__iand__(self, other) object.__ixor__(self, other) object.__ior__(self, other) These methods are called to implement the augmented arithmetic assignments (``+=``, ``-=``, ``*=``, ``@=``, ``/=``, ``//=``, ``%=``, ``**=``, ``<<=``, ``>>=``, ``&=``, ``^=``, ``|=``). These methods should attempt to do the operation in-place (modifying *self*) and return the result (which could be, but does not have to be, *self*). If a specific method is not defined, the augmented assignment falls back to the normal methods. For instance, if *x* is an instance of a class with an :meth:`__iadd__` method, ``x += y`` is equivalent to ``x = x.__iadd__(y)`` . Otherwise, ``x.__add__(y)`` and ``y.__radd__(x)`` are considered, as with the evaluation of ``x + y``. In certain situations, augmented assignment can result in unexpected errors (see :ref:`faq-augmented-assignment-tuple-error`), but this behavior is in fact part of the data model. .. method:: object.__neg__(self) object.__pos__(self) object.__abs__(self) object.__invert__(self) .. index:: builtin: abs Called to implement the unary arithmetic operations (``-``, ``+``, :func:`abs` and ``~``). .. method:: object.__complex__(self) object.__int__(self) object.__float__(self) .. index:: builtin: complex builtin: int builtin: float Called to implement the built-in functions :func:`complex`, :func:`int` and :func:`float`. Should return a value of the appropriate type. .. method:: object.__index__(self) Called to implement :func:`operator.index`, and whenever Python needs to losslessly convert the numeric object to an integer object (such as in slicing, or in the built-in :func:`bin`, :func:`hex` and :func:`oct` functions). Presence of this method indicates that the numeric object is an integer type. Must return an integer. .. note:: In order to have a coherent integer type class, when :meth:`__index__` is defined :meth:`__int__` should also be defined, and both should return the same value. .. method:: object.__round__(self, [,ndigits]) object.__trunc__(self) object.__floor__(self) object.__ceil__(self) .. index:: builtin: round Called to implement the built-in function :func:`round` and :mod:`math` functions :func:`~math.trunc`, :func:`~math.floor` and :func:`~math.ceil`. Unless *ndigits* is passed to :meth:`!__round__` all these methods should return the value of the object truncated to an :class:`~numbers.Integral` (typically an :class:`int`). If :meth:`__int__` is not defined then the built-in function :func:`int` falls back to :meth:`__trunc__`. .. _context-managers: With Statement Context Managers ------------------------------- A :dfn:`context manager` is an object that defines the runtime context to be established when executing a :keyword:`with` statement. The context manager handles the entry into, and the exit from, the desired runtime context for the execution of the block of code. Context managers are normally invoked using the :keyword:`with` statement (described in section :ref:`with`), but can also be used by directly invoking their methods. .. index:: statement: with single: context manager Typical uses of context managers include saving and restoring various kinds of global state, locking and unlocking resources, closing opened files, etc. For more information on context managers, see :ref:`typecontextmanager`. .. method:: object.__enter__(self) Enter the runtime context related to this object. The :keyword:`with` statement will bind this method's return value to the target(s) specified in the :keyword:`as` clause of the statement, if any. .. method:: object.__exit__(self, exc_type, exc_value, traceback) Exit the runtime context related to this object. The parameters describe the exception that caused the context to be exited. If the context was exited without an exception, all three arguments will be :const:`None`. If an exception is supplied, and the method wishes to suppress the exception (i.e., prevent it from being propagated), it should return a true value. Otherwise, the exception will be processed normally upon exit from this method. Note that :meth:`__exit__` methods should not reraise the passed-in exception; this is the caller's responsibility. .. seealso:: :pep:`343` - The "with" statement The specification, background, and examples for the Python :keyword:`with` statement. .. _special-lookup: Special method lookup --------------------- For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object's type, not in the object's instance dictionary. That behaviour is the reason why the following code raises an exception:: >>> class C: ... pass ... >>> c = C() >>> c.__len__ = lambda: 5 >>> len(c) Traceback (most recent call last): File "", line 1, in TypeError: object of type 'C' has no len() The rationale behind this behaviour lies with a number of special methods such as :meth:`__hash__` and :meth:`__repr__` that are implemented by all objects, including type objects. If the implicit lookup of these methods used the conventional lookup process, they would fail when invoked on the type object itself:: >>> 1 .__hash__() == hash(1) True >>> int.__hash__() == hash(int) Traceback (most recent call last): File "", line 1, in TypeError: descriptor '__hash__' of 'int' object needs an argument Incorrectly attempting to invoke an unbound method of a class in this way is sometimes referred to as 'metaclass confusion', and is avoided by bypassing the instance when looking up special methods:: >>> type(1).__hash__(1) == hash(1) True >>> type(int).__hash__(int) == hash(int) True In addition to bypassing any instance attributes in the interest of correctness, implicit special method lookup generally also bypasses the :meth:`__getattribute__` method even of the object's metaclass:: >>> class Meta(type): ... def __getattribute__(*args): ... print("Metaclass getattribute invoked") ... return type.__getattribute__(*args) ... >>> class C(object, metaclass=Meta): ... def __len__(self): ... return 10 ... def __getattribute__(*args): ... print("Class getattribute invoked") ... return object.__getattribute__(*args) ... >>> c = C() >>> c.__len__() # Explicit lookup via instance Class getattribute invoked 10 >>> type(c).__len__(c) # Explicit lookup via type Metaclass getattribute invoked 10 >>> len(c) # Implicit lookup 10 Bypassing the :meth:`__getattribute__` machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods (the special method *must* be set on the class object itself in order to be consistently invoked by the interpreter). .. index:: single: coroutine Coroutines ========== Awaitable Objects ----------------- An :term:`awaitable` object generally implements an :meth:`__await__` method. :term:`Coroutine` objects returned from :keyword:`async def` functions are awaitable. .. note:: The :term:`generator iterator` objects returned from generators decorated with :func:`types.coroutine` or :func:`asyncio.coroutine` are also awaitable, but they do not implement :meth:`__await__`. .. method:: object.__await__(self) Must return an :term:`iterator`. Should be used to implement :term:`awaitable` objects. For instance, :class:`asyncio.Future` implements this method to be compatible with the :keyword:`await` expression. .. versionadded:: 3.5 .. seealso:: :pep:`492` for additional information about awaitable objects. .. _coroutine-objects: Coroutine Objects ----------------- :term:`Coroutine` objects are :term:`awaitable` objects. A coroutine's execution can be controlled by calling :meth:`__await__` and iterating over the result. When the coroutine has finished executing and returns, the iterator raises :exc:`StopIteration`, and the exception's :attr:`~StopIteration.value` attribute holds the return value. If the coroutine raises an exception, it is propagated by the iterator. Coroutines should not directly raise unhandled :exc:`StopIteration` exceptions. Coroutines also have the methods listed below, which are analogous to those of generators (see :ref:`generator-methods`). However, unlike generators, coroutines do not directly support iteration. .. versionchanged:: 3.5.2 It is a :exc:`RuntimeError` to await on a coroutine more than once. .. method:: coroutine.send(value) Starts or resumes execution of the coroutine. If *value* is ``None``, this is equivalent to advancing the iterator returned by :meth:`__await__`. If *value* is not ``None``, this method delegates to the :meth:`~generator.send` method of the iterator that caused the coroutine to suspend. The result (return value, :exc:`StopIteration`, or other exception) is the same as when iterating over the :meth:`__await__` return value, described above. .. method:: coroutine.throw(type[, value[, traceback]]) Raises the specified exception in the coroutine. This method delegates to the :meth:`~generator.throw` method of the iterator that caused the coroutine to suspend, if it has such a method. Otherwise, the exception is raised at the suspension point. The result (return value, :exc:`StopIteration`, or other exception) is the same as when iterating over the :meth:`__await__` return value, described above. If the exception is not caught in the coroutine, it propagates back to the caller. .. method:: coroutine.close() Causes the coroutine to clean itself up and exit. If the coroutine is suspended, this method first delegates to the :meth:`~generator.close` method of the iterator that caused the coroutine to suspend, if it has such a method. Then it raises :exc:`GeneratorExit` at the suspension point, causing the coroutine to immediately clean itself up. Finally, the coroutine is marked as having finished executing, even if it was never started. Coroutine objects are automatically closed using the above process when they are about to be destroyed. .. _async-iterators: Asynchronous Iterators ---------------------- An *asynchronous iterable* is able to call asynchronous code in its ``__aiter__`` implementation, and an *asynchronous iterator* can call asynchronous code in its ``__anext__`` method. Asynchronous iterators can be used in an :keyword:`async for` statement. .. method:: object.__aiter__(self) Must return an *asynchronous iterator* object. .. method:: object.__anext__(self) Must return an *awaitable* resulting in a next value of the iterator. Should raise a :exc:`StopAsyncIteration` error when the iteration is over. An example of an asynchronous iterable object:: class Reader: async def readline(self): ... def __aiter__(self): return self async def __anext__(self): val = await self.readline() if val == b'': raise StopAsyncIteration return val .. versionadded:: 3.5 .. note:: .. versionchanged:: 3.5.2 Starting with CPython 3.5.2, ``__aiter__`` can directly return :term:`asynchronous iterators `. Returning an :term:`awaitable` object will result in a :exc:`PendingDeprecationWarning`. The recommended way of writing backwards compatible code in CPython 3.5.x is to continue returning awaitables from ``__aiter__``. If you want to avoid the PendingDeprecationWarning and keep the code backwards compatible, the following decorator can be used:: import functools import sys if sys.version_info < (3, 5, 2): def aiter_compat(func): @functools.wraps(func) async def wrapper(self): return func(self) return wrapper else: def aiter_compat(func): return func Example:: class AsyncIterator: @aiter_compat def __aiter__(self): return self async def __anext__(self): ... Starting with CPython 3.6, the :exc:`PendingDeprecationWarning` will be replaced with the :exc:`DeprecationWarning`. In CPython 3.7, returning an awaitable from ``__aiter__`` will result in a :exc:`RuntimeError`. Asynchronous Context Managers ----------------------------- An *asynchronous context manager* is a *context manager* that is able to suspend execution in its ``__aenter__`` and ``__aexit__`` methods. Asynchronous context managers can be used in an :keyword:`async with` statement. .. method:: object.__aenter__(self) This method is semantically similar to the :meth:`__enter__`, with only difference that it must return an *awaitable*. .. method:: object.__aexit__(self, exc_type, exc_value, traceback) This method is semantically similar to the :meth:`__exit__`, with only difference that it must return an *awaitable*. An example of an asynchronous context manager class:: class AsyncContextManager: async def __aenter__(self): await log('entering context') async def __aexit__(self, exc_type, exc, tb): await log('exiting context') .. versionadded:: 3.5 .. rubric:: Footnotes .. [#] It *is* possible in some cases to change an object's type, under certain controlled conditions. It generally isn't a good idea though, since it can lead to some very strange behaviour if it is handled incorrectly. .. [#] The :meth:`__hash__`, :meth:`__iter__`, :meth:`__reversed__`, and :meth:`__contains__` methods have special handling for this; others will still raise a :exc:`TypeError`, but may do so by relying on the behavior that ``None`` is not callable. .. [#] "Does not support" here means that the class has no such method, or the method returns ``NotImplemented``. Do not set the method to ``None`` if you want to force fallback to the right operand's reflected method—that will instead have the opposite effect of explicitly *blocking* such fallback. .. [#] For operands of the same type, it is assumed that if the non-reflected method (such as :meth:`__add__`) fails the operation is not supported, which is why the reflected method is not called. PK 3]F0qq reference/compound_stmts.rst.txtnu[.. _compound: ******************* Compound statements ******************* .. index:: pair: compound; statement Compound statements contain (groups of) other statements; they affect or control the execution of those other statements in some way. In general, compound statements span multiple lines, although in simple incarnations a whole compound statement may be contained in one line. The :keyword:`if`, :keyword:`while` and :keyword:`for` statements implement traditional control flow constructs. :keyword:`try` specifies exception handlers and/or cleanup code for a group of statements, while the :keyword:`with` statement allows the execution of initialization and finalization code around a block of code. Function and class definitions are also syntactically compound statements. .. index:: single: clause single: suite A compound statement consists of one or more 'clauses.' A clause consists of a header and a 'suite.' The clause headers of a particular compound statement are all at the same indentation level. Each clause header begins with a uniquely identifying keyword and ends with a colon. A suite is a group of statements controlled by a clause. A suite can be one or more semicolon-separated simple statements on the same line as the header, following the header's colon, or it can be one or more indented statements on subsequent lines. Only the latter form of a suite can contain nested compound statements; the following is illegal, mostly because it wouldn't be clear to which :keyword:`if` clause a following :keyword:`else` clause would belong:: if test1: if test2: print(x) Also note that the semicolon binds tighter than the colon in this context, so that in the following example, either all or none of the :func:`print` calls are executed:: if x < y < z: print(x); print(y); print(z) Summarizing: .. productionlist:: compound_stmt: `if_stmt` : | `while_stmt` : | `for_stmt` : | `try_stmt` : | `with_stmt` : | `funcdef` : | `classdef` : | `async_with_stmt` : | `async_for_stmt` : | `async_funcdef` suite: `stmt_list` NEWLINE | NEWLINE INDENT `statement`+ DEDENT statement: `stmt_list` NEWLINE | `compound_stmt` stmt_list: `simple_stmt` (";" `simple_stmt`)* [";"] .. index:: single: NEWLINE token single: DEDENT token pair: dangling; else Note that statements always end in a ``NEWLINE`` possibly followed by a ``DEDENT``. Also note that optional continuation clauses always begin with a keyword that cannot start a statement, thus there are no ambiguities (the 'dangling :keyword:`else`' problem is solved in Python by requiring nested :keyword:`if` statements to be indented). The formatting of the grammar rules in the following sections places each clause on a separate line for clarity. .. _if: .. _elif: .. _else: The :keyword:`if` statement =========================== .. index:: statement: if keyword: elif keyword: else keyword: elif keyword: else The :keyword:`if` statement is used for conditional execution: .. productionlist:: if_stmt: "if" `expression` ":" `suite` : ("elif" `expression` ":" `suite`)* : ["else" ":" `suite`] It selects exactly one of the suites by evaluating the expressions one by one until one is found to be true (see section :ref:`booleans` for the definition of true and false); then that suite is executed (and no other part of the :keyword:`if` statement is executed or evaluated). If all expressions are false, the suite of the :keyword:`else` clause, if present, is executed. .. _while: The :keyword:`while` statement ============================== .. index:: statement: while keyword: else pair: loop; statement keyword: else The :keyword:`while` statement is used for repeated execution as long as an expression is true: .. productionlist:: while_stmt: "while" `expression` ":" `suite` : ["else" ":" `suite`] This repeatedly tests the expression and, if it is true, executes the first suite; if the expression is false (which may be the first time it is tested) the suite of the :keyword:`else` clause, if present, is executed and the loop terminates. .. index:: statement: break statement: continue A :keyword:`break` statement executed in the first suite terminates the loop without executing the :keyword:`else` clause's suite. A :keyword:`continue` statement executed in the first suite skips the rest of the suite and goes back to testing the expression. .. _for: The :keyword:`for` statement ============================ .. index:: statement: for keyword: in keyword: else pair: target; list pair: loop; statement keyword: in keyword: else pair: target; list object: sequence The :keyword:`for` statement is used to iterate over the elements of a sequence (such as a string, tuple or list) or other iterable object: .. productionlist:: for_stmt: "for" `target_list` "in" `expression_list` ":" `suite` : ["else" ":" `suite`] The expression list is evaluated once; it should yield an iterable object. An iterator is created for the result of the ``expression_list``. The suite is then executed once for each item provided by the iterator, in the order returned by the iterator. Each item in turn is assigned to the target list using the standard rules for assignments (see :ref:`assignment`), and then the suite is executed. When the items are exhausted (which is immediately when the sequence is empty or an iterator raises a :exc:`StopIteration` exception), the suite in the :keyword:`else` clause, if present, is executed, and the loop terminates. .. index:: statement: break statement: continue A :keyword:`break` statement executed in the first suite terminates the loop without executing the :keyword:`else` clause's suite. A :keyword:`continue` statement executed in the first suite skips the rest of the suite and continues with the next item, or with the :keyword:`else` clause if there is no next item. The for-loop makes assignments to the variables(s) in the target list. This overwrites all previous assignments to those variables including those made in the suite of the for-loop:: for i in range(10): print(i) i = 5 # this will not affect the for-loop # because i will be overwritten with the next # index in the range .. index:: builtin: range Names in the target list are not deleted when the loop is finished, but if the sequence is empty, they will not have been assigned to at all by the loop. Hint: the built-in function :func:`range` returns an iterator of integers suitable to emulate the effect of Pascal's ``for i := a to b do``; e.g., ``list(range(3))`` returns the list ``[0, 1, 2]``. .. note:: .. index:: single: loop; over mutable sequence single: mutable sequence; loop over There is a subtlety when the sequence is being modified by the loop (this can only occur for mutable sequences, e.g. lists). An internal counter is used to keep track of which item is used next, and this is incremented on each iteration. When this counter has reached the length of the sequence the loop terminates. This means that if the suite deletes the current (or a previous) item from the sequence, the next item will be skipped (since it gets the index of the current item which has already been treated). Likewise, if the suite inserts an item in the sequence before the current item, the current item will be treated again the next time through the loop. This can lead to nasty bugs that can be avoided by making a temporary copy using a slice of the whole sequence, e.g., :: for x in a[:]: if x < 0: a.remove(x) .. _try: .. _except: .. _finally: The :keyword:`try` statement ============================ .. index:: statement: try keyword: except keyword: finally .. index:: keyword: except The :keyword:`try` statement specifies exception handlers and/or cleanup code for a group of statements: .. productionlist:: try_stmt: `try1_stmt` | `try2_stmt` try1_stmt: "try" ":" `suite` : ("except" [`expression` ["as" `identifier`]] ":" `suite`)+ : ["else" ":" `suite`] : ["finally" ":" `suite`] try2_stmt: "try" ":" `suite` : "finally" ":" `suite` The :keyword:`except` clause(s) specify one or more exception handlers. When no exception occurs in the :keyword:`try` clause, no exception handler is executed. When an exception occurs in the :keyword:`try` suite, a search for an exception handler is started. This search inspects the except clauses in turn until one is found that matches the exception. An expression-less except clause, if present, must be last; it matches any exception. For an except clause with an expression, that expression is evaluated, and the clause matches the exception if the resulting object is "compatible" with the exception. An object is compatible with an exception if it is the class or a base class of the exception object or a tuple containing an item compatible with the exception. If no except clause matches the exception, the search for an exception handler continues in the surrounding code and on the invocation stack. [#]_ If the evaluation of an expression in the header of an except clause raises an exception, the original search for a handler is canceled and a search starts for the new exception in the surrounding code and on the call stack (it is treated as if the entire :keyword:`try` statement raised the exception). When a matching except clause is found, the exception is assigned to the target specified after the :keyword:`as` keyword in that except clause, if present, and the except clause's suite is executed. All except clauses must have an executable block. When the end of this block is reached, execution continues normally after the entire try statement. (This means that if two nested handlers exist for the same exception, and the exception occurs in the try clause of the inner handler, the outer handler will not handle the exception.) When an exception has been assigned using ``as target``, it is cleared at the end of the except clause. This is as if :: except E as N: foo was translated to :: except E as N: try: foo finally: del N This means the exception must be assigned to a different name to be able to refer to it after the except clause. Exceptions are cleared because with the traceback attached to them, they form a reference cycle with the stack frame, keeping all locals in that frame alive until the next garbage collection occurs. .. index:: module: sys object: traceback Before an except clause's suite is executed, details about the exception are stored in the :mod:`sys` module and can be accessed via :func:`sys.exc_info`. :func:`sys.exc_info` returns a 3-tuple consisting of the exception class, the exception instance and a traceback object (see section :ref:`types`) identifying the point in the program where the exception occurred. :func:`sys.exc_info` values are restored to their previous values (before the call) when returning from a function that handled an exception. .. index:: keyword: else statement: return statement: break statement: continue The optional :keyword:`else` clause is executed if and when control flows off the end of the :keyword:`try` clause. [#]_ Exceptions in the :keyword:`else` clause are not handled by the preceding :keyword:`except` clauses. .. index:: keyword: finally If :keyword:`finally` is present, it specifies a 'cleanup' handler. The :keyword:`try` clause is executed, including any :keyword:`except` and :keyword:`else` clauses. If an exception occurs in any of the clauses and is not handled, the exception is temporarily saved. The :keyword:`finally` clause is executed. If there is a saved exception it is re-raised at the end of the :keyword:`finally` clause. If the :keyword:`finally` clause raises another exception, the saved exception is set as the context of the new exception. If the :keyword:`finally` clause executes a :keyword:`return` or :keyword:`break` statement, the saved exception is discarded:: >>> def f(): ... try: ... 1/0 ... finally: ... return 42 ... >>> f() 42 The exception information is not available to the program during execution of the :keyword:`finally` clause. .. index:: statement: return statement: break statement: continue When a :keyword:`return`, :keyword:`break` or :keyword:`continue` statement is executed in the :keyword:`try` suite of a :keyword:`try`...\ :keyword:`finally` statement, the :keyword:`finally` clause is also executed 'on the way out.' A :keyword:`continue` statement is illegal in the :keyword:`finally` clause. (The reason is a problem with the current implementation --- this restriction may be lifted in the future). The return value of a function is determined by the last :keyword:`return` statement executed. Since the :keyword:`finally` clause always executes, a :keyword:`return` statement executed in the :keyword:`finally` clause will always be the last one executed:: >>> def foo(): ... try: ... return 'try' ... finally: ... return 'finally' ... >>> foo() 'finally' Additional information on exceptions can be found in section :ref:`exceptions`, and information on using the :keyword:`raise` statement to generate exceptions may be found in section :ref:`raise`. .. _with: .. _as: The :keyword:`with` statement ============================= .. index:: statement: with single: as; with statement The :keyword:`with` statement is used to wrap the execution of a block with methods defined by a context manager (see section :ref:`context-managers`). This allows common :keyword:`try`...\ :keyword:`except`...\ :keyword:`finally` usage patterns to be encapsulated for convenient reuse. .. productionlist:: with_stmt: "with" `with_item` ("," `with_item`)* ":" `suite` with_item: `expression` ["as" `target`] The execution of the :keyword:`with` statement with one "item" proceeds as follows: #. The context expression (the expression given in the :token:`with_item`) is evaluated to obtain a context manager. #. The context manager's :meth:`__exit__` is loaded for later use. #. The context manager's :meth:`__enter__` method is invoked. #. If a target was included in the :keyword:`with` statement, the return value from :meth:`__enter__` is assigned to it. .. note:: The :keyword:`with` statement guarantees that if the :meth:`__enter__` method returns without an error, then :meth:`__exit__` will always be called. Thus, if an error occurs during the assignment to the target list, it will be treated the same as an error occurring within the suite would be. See step 6 below. #. The suite is executed. #. The context manager's :meth:`__exit__` method is invoked. If an exception caused the suite to be exited, its type, value, and traceback are passed as arguments to :meth:`__exit__`. Otherwise, three :const:`None` arguments are supplied. If the suite was exited due to an exception, and the return value from the :meth:`__exit__` method was false, the exception is reraised. If the return value was true, the exception is suppressed, and execution continues with the statement following the :keyword:`with` statement. If the suite was exited for any reason other than an exception, the return value from :meth:`__exit__` is ignored, and execution proceeds at the normal location for the kind of exit that was taken. With more than one item, the context managers are processed as if multiple :keyword:`with` statements were nested:: with A() as a, B() as b: suite is equivalent to :: with A() as a: with B() as b: suite .. versionchanged:: 3.1 Support for multiple context expressions. .. seealso:: :pep:`343` - The "with" statement The specification, background, and examples for the Python :keyword:`with` statement. .. index:: single: parameter; function definition .. _function: .. _def: Function definitions ==================== .. index:: statement: def pair: function; definition pair: function; name pair: name; binding object: user-defined function object: function pair: function; name pair: name; binding A function definition defines a user-defined function object (see section :ref:`types`): .. productionlist:: funcdef: [`decorators`] "def" `funcname` "(" [`parameter_list`] ")" : ["->" `expression`] ":" `suite` decorators: `decorator`+ decorator: "@" `dotted_name` ["(" [`argument_list` [","]] ")"] NEWLINE dotted_name: `identifier` ("." `identifier`)* parameter_list: `defparameter` ("," `defparameter`)* ["," [`parameter_list_starargs`]] : | `parameter_list_starargs` parameter_list_starargs: "*" [`parameter`] ("," `defparameter`)* ["," ["**" `parameter` [","]]] : | "**" `parameter` [","] parameter: `identifier` [":" `expression`] defparameter: `parameter` ["=" `expression`] funcname: `identifier` A function definition is an executable statement. Its execution binds the function name in the current local namespace to a function object (a wrapper around the executable code for the function). This function object contains a reference to the current global namespace as the global namespace to be used when the function is called. The function definition does not execute the function body; this gets executed only when the function is called. [#]_ .. index:: statement: @ A function definition may be wrapped by one or more :term:`decorator` expressions. Decorator expressions are evaluated when the function is defined, in the scope that contains the function definition. The result must be a callable, which is invoked with the function object as the only argument. The returned value is bound to the function name instead of the function object. Multiple decorators are applied in nested fashion. For example, the following code :: @f1(arg) @f2 def func(): pass is roughly equivalent to :: def func(): pass func = f1(arg)(f2(func)) except that the original function is not temporarily bound to the name ``func``. .. index:: triple: default; parameter; value single: argument; function definition When one or more :term:`parameters ` have the form *parameter* ``=`` *expression*, the function is said to have "default parameter values." For a parameter with a default value, the corresponding :term:`argument` may be omitted from a call, in which case the parameter's default value is substituted. If a parameter has a default value, all following parameters up until the "``*``" must also have a default value --- this is a syntactic restriction that is not expressed by the grammar. **Default parameter values are evaluated from left to right when the function definition is executed.** This means that the expression is evaluated once, when the function is defined, and that the same "pre-computed" value is used for each call. This is especially important to understand when a default parameter is a mutable object, such as a list or a dictionary: if the function modifies the object (e.g. by appending an item to a list), the default value is in effect modified. This is generally not what was intended. A way around this is to use ``None`` as the default, and explicitly test for it in the body of the function, e.g.:: def whats_on_the_telly(penguin=None): if penguin is None: penguin = [] penguin.append("property of the zoo") return penguin .. index:: statement: * statement: ** Function call semantics are described in more detail in section :ref:`calls`. A function call always assigns values to all parameters mentioned in the parameter list, either from position arguments, from keyword arguments, or from default values. If the form "``*identifier``" is present, it is initialized to a tuple receiving any excess positional parameters, defaulting to the empty tuple. If the form "``**identifier``" is present, it is initialized to a new ordered mapping receiving any excess keyword arguments, defaulting to a new empty mapping of the same type. Parameters after "``*``" or "``*identifier``" are keyword-only parameters and may only be passed used keyword arguments. .. index:: pair: function; annotations Parameters may have annotations of the form "``: expression``" following the parameter name. Any parameter may have an annotation even those of the form ``*identifier`` or ``**identifier``. Functions may have "return" annotation of the form "``-> expression``" after the parameter list. These annotations can be any valid Python expression and are evaluated when the function definition is executed. Annotations may be evaluated in a different order than they appear in the source code. The presence of annotations does not change the semantics of a function. The annotation values are available as values of a dictionary keyed by the parameters' names in the :attr:`__annotations__` attribute of the function object. .. index:: pair: lambda; expression It is also possible to create anonymous functions (functions not bound to a name), for immediate use in expressions. This uses lambda expressions, described in section :ref:`lambda`. Note that the lambda expression is merely a shorthand for a simplified function definition; a function defined in a ":keyword:`def`" statement can be passed around or assigned to another name just like a function defined by a lambda expression. The ":keyword:`def`" form is actually more powerful since it allows the execution of multiple statements and annotations. **Programmer's note:** Functions are first-class objects. A "``def``" statement executed inside a function definition defines a local function that can be returned or passed around. Free variables used in the nested function can access the local variables of the function containing the def. See section :ref:`naming` for details. .. seealso:: :pep:`3107` - Function Annotations The original specification for function annotations. .. _class: Class definitions ================= .. index:: object: class statement: class pair: class; definition pair: class; name pair: name; binding pair: execution; frame single: inheritance single: docstring A class definition defines a class object (see section :ref:`types`): .. productionlist:: classdef: [`decorators`] "class" `classname` [`inheritance`] ":" `suite` inheritance: "(" [`argument_list`] ")" classname: `identifier` A class definition is an executable statement. The inheritance list usually gives a list of base classes (see :ref:`metaclasses` for more advanced uses), so each item in the list should evaluate to a class object which allows subclassing. Classes without an inheritance list inherit, by default, from the base class :class:`object`; hence, :: class Foo: pass is equivalent to :: class Foo(object): pass The class's suite is then executed in a new execution frame (see :ref:`naming`), using a newly created local namespace and the original global namespace. (Usually, the suite contains mostly function definitions.) When the class's suite finishes execution, its execution frame is discarded but its local namespace is saved. [#]_ A class object is then created using the inheritance list for the base classes and the saved local namespace for the attribute dictionary. The class name is bound to this class object in the original local namespace. The order in which attributes are defined in the class body is preserved in the new class's ``__dict__``. Note that this is reliable only right after the class is created and only for classes that were defined using the definition syntax. Class creation can be customized heavily using :ref:`metaclasses `. Classes can also be decorated: just like when decorating functions, :: @f1(arg) @f2 class Foo: pass is roughly equivalent to :: class Foo: pass Foo = f1(arg)(f2(Foo)) The evaluation rules for the decorator expressions are the same as for function decorators. The result is then bound to the class name. **Programmer's note:** Variables defined in the class definition are class attributes; they are shared by instances. Instance attributes can be set in a method with ``self.name = value``. Both class and instance attributes are accessible through the notation "``self.name``", and an instance attribute hides a class attribute with the same name when accessed in this way. Class attributes can be used as defaults for instance attributes, but using mutable values there can lead to unexpected results. :ref:`Descriptors ` can be used to create instance variables with different implementation details. .. seealso:: :pep:`3115` - Metaclasses in Python 3 :pep:`3129` - Class Decorators Coroutines ========== .. versionadded:: 3.5 .. index:: statement: async def .. _`async def`: Coroutine function definition ----------------------------- .. productionlist:: async_funcdef: [`decorators`] "async" "def" `funcname` "(" [`parameter_list`] ")" : ["->" `expression`] ":" `suite` .. index:: keyword: async keyword: await Execution of Python coroutines can be suspended and resumed at many points (see :term:`coroutine`). In the body of a coroutine, any ``await`` and ``async`` identifiers become reserved keywords; :keyword:`await` expressions, :keyword:`async for` and :keyword:`async with` can only be used in coroutine bodies. Functions defined with ``async def`` syntax are always coroutine functions, even if they do not contain ``await`` or ``async`` keywords. It is a :exc:`SyntaxError` to use ``yield from`` expressions in ``async def`` coroutines. An example of a coroutine function:: async def func(param1, param2): do_stuff() await some_coroutine() .. index:: statement: async for .. _`async for`: The :keyword:`async for` statement ---------------------------------- .. productionlist:: async_for_stmt: "async" `for_stmt` An :term:`asynchronous iterable` is able to call asynchronous code in its *iter* implementation, and :term:`asynchronous iterator` can call asynchronous code in its *next* method. The ``async for`` statement allows convenient iteration over asynchronous iterators. The following code:: async for TARGET in ITER: BLOCK else: BLOCK2 Is semantically equivalent to:: iter = (ITER) iter = type(iter).__aiter__(iter) running = True while running: try: TARGET = await type(iter).__anext__(iter) except StopAsyncIteration: running = False else: BLOCK else: BLOCK2 See also :meth:`__aiter__` and :meth:`__anext__` for details. It is a :exc:`SyntaxError` to use ``async for`` statement outside of an :keyword:`async def` function. .. index:: statement: async with .. _`async with`: The :keyword:`async with` statement ----------------------------------- .. productionlist:: async_with_stmt: "async" `with_stmt` An :term:`asynchronous context manager` is a :term:`context manager` that is able to suspend execution in its *enter* and *exit* methods. The following code:: async with EXPR as VAR: BLOCK Is semantically equivalent to:: mgr = (EXPR) aexit = type(mgr).__aexit__ aenter = type(mgr).__aenter__(mgr) VAR = await aenter try: BLOCK except: if not await aexit(mgr, *sys.exc_info()): raise else: await aexit(mgr, None, None, None) See also :meth:`__aenter__` and :meth:`__aexit__` for details. It is a :exc:`SyntaxError` to use ``async with`` statement outside of an :keyword:`async def` function. .. seealso:: :pep:`492` - Coroutines with async and await syntax .. rubric:: Footnotes .. [#] The exception is propagated to the invocation stack unless there is a :keyword:`finally` clause which happens to raise another exception. That new exception causes the old one to be lost. .. [#] Currently, control "flows off the end" except in the case of an exception or the execution of a :keyword:`return`, :keyword:`continue`, or :keyword:`break` statement. .. [#] A string literal appearing as the first statement in the function body is transformed into the function's ``__doc__`` attribute and therefore the function's :term:`docstring`. .. [#] A string literal appearing as the first statement in the class body is transformed into the namespace's ``__doc__`` item and therefore the class's :term:`docstring`. PK 3]JNW%%installing/index.rst.txtnu[.. highlightlang:: none .. _installing-index: ************************* Installing Python Modules ************************* :Email: distutils-sig@python.org As a popular open source development project, Python has an active supporting community of contributors and users that also make their software available for other Python developers to use under open source license terms. This allows Python users to share and collaborate effectively, benefiting from the solutions others have already created to common (and sometimes even rare!) problems, as well as potentially contributing their own solutions to the common pool. This guide covers the installation part of the process. For a guide to creating and sharing your own Python projects, refer to the :ref:`distribution guide `. .. note:: For corporate and other institutional users, be aware that many organisations have their own policies around using and contributing to open source software. Please take such policies into account when making use of the distribution and installation tools provided with Python. Key terms ========= * ``pip`` is the preferred installer program. Starting with Python 3.4, it is included by default with the Python binary installers. * A *virtual environment* is a semi-isolated Python environment that allows packages to be installed for use by a particular application, rather than being installed system wide. * ``venv`` is the standard tool for creating virtual environments, and has been part of Python since Python 3.3. Starting with Python 3.4, it defaults to installing ``pip`` into all created virtual environments. * ``virtualenv`` is a third party alternative (and predecessor) to ``venv``. It allows virtual environments to be used on versions of Python prior to 3.4, which either don't provide ``venv`` at all, or aren't able to automatically install ``pip`` into created environments. * The `Python Packaging Index `__ is a public repository of open source licensed packages made available for use by other Python users. * the `Python Packaging Authority `__ are the group of developers and documentation authors responsible for the maintenance and evolution of the standard packaging tools and the associated metadata and file format standards. They maintain a variety of tools, documentation, and issue trackers on both `GitHub `__ and `BitBucket `__. * ``distutils`` is the original build and distribution system first added to the Python standard library in 1998. While direct use of ``distutils`` is being phased out, it still laid the foundation for the current packaging and distribution infrastructure, and it not only remains part of the standard library, but its name lives on in other ways (such as the name of the mailing list used to coordinate Python packaging standards development). .. deprecated:: 3.6 ``pyvenv`` was the recommended tool for creating virtual environments for Python 3.3 and 3.4, and is `deprecated in Python 3.6 `_. .. versionchanged:: 3.5 The use of ``venv`` is now recommended for creating virtual environments. .. seealso:: `Python Packaging User Guide: Creating and using virtual environments `__ Basic usage =========== The standard packaging tools are all designed to be used from the command line. The following command will install the latest version of a module and its dependencies from the Python Packaging Index:: python -m pip install SomePackage .. note:: For POSIX users (including Mac OS X and Linux users), the examples in this guide assume the use of a :term:`virtual environment`. For Windows users, the examples in this guide assume that the option to adjust the system PATH environment variable was selected when installing Python. It's also possible to specify an exact or minimum version directly on the command line. When using comparator operators such as ``>``, ``<`` or some other special character which get interpreted by shell, the package name and the version should be enclosed within double quotes:: python -m pip install SomePackage==1.0.4 # specific version python -m pip install "SomePackage>=1.0.4" # minimum version Normally, if a suitable module is already installed, attempting to install it again will have no effect. Upgrading existing modules must be requested explicitly:: python -m pip install --upgrade SomePackage More information and resources regarding ``pip`` and its capabilities can be found in the `Python Packaging User Guide `__. Creation of virtual environments is done through the :mod:`venv` module. Installing packages into an active virtual environment uses the commands shown above. .. seealso:: `Python Packaging User Guide: Installing Python Distribution Packages `__ How do I ...? ============= These are quick answers or links for some common tasks. ... install ``pip`` in versions of Python prior to Python 3.4? -------------------------------------------------------------- Python only started bundling ``pip`` with Python 3.4. For earlier versions, ``pip`` needs to be "bootstrapped" as described in the Python Packaging User Guide. .. seealso:: `Python Packaging User Guide: Requirements for Installing Packages `__ .. installing-per-user-installation: ... install packages just for the current user? ----------------------------------------------- Passing the ``--user`` option to ``python -m pip install`` will install a package just for the current user, rather than for all users of the system. ... install scientific Python packages? --------------------------------------- A number of scientific Python packages have complex binary dependencies, and aren't currently easy to install using ``pip`` directly. At this point in time, it will often be easier for users to install these packages by `other means `__ rather than attempting to install them with ``pip``. .. seealso:: `Python Packaging User Guide: Installing Scientific Packages `__ ... work with multiple versions of Python installed in parallel? ---------------------------------------------------------------- On Linux, Mac OS X, and other POSIX systems, use the versioned Python commands in combination with the ``-m`` switch to run the appropriate copy of ``pip``:: python2 -m pip install SomePackage # default Python 2 python2.7 -m pip install SomePackage # specifically Python 2.7 python3 -m pip install SomePackage # default Python 3 python3.4 -m pip install SomePackage # specifically Python 3.4 Appropriately versioned ``pip`` commands may also be available. On Windows, use the ``py`` Python launcher in combination with the ``-m`` switch:: py -2 -m pip install SomePackage # default Python 2 py -2.7 -m pip install SomePackage # specifically Python 2.7 py -3 -m pip install SomePackage # default Python 3 py -3.4 -m pip install SomePackage # specifically Python 3.4 .. other questions: Once the Development & Deployment part of PPUG is fleshed out, some of those sections should be linked from new questions here (most notably, we should have a question about avoiding depending on PyPI that links to https://packaging.python.org/en/latest/mirrors/) Common installation issues ========================== Installing into the system Python on Linux ------------------------------------------ On Linux systems, a Python installation will typically be included as part of the distribution. Installing into this Python installation requires root access to the system, and may interfere with the operation of the system package manager and other components of the system if a component is unexpectedly upgraded using ``pip``. On such systems, it is often better to use a virtual environment or a per-user installation when installing packages with ``pip``. Pip not installed ----------------- It is possible that ``pip`` does not get installed by default. One potential fix is:: python -m ensurepip --default-pip There are also additional resources for `installing pip. `__ Installing binary extensions ---------------------------- Python has typically relied heavily on source based distribution, with end users being expected to compile extension modules from source as part of the installation process. With the introduction of support for the binary ``wheel`` format, and the ability to publish wheels for at least Windows and Mac OS X through the Python Packaging Index, this problem is expected to diminish over time, as users are more regularly able to install pre-built extensions rather than needing to build them themselves. Some of the solutions for installing `scientific software `__ that are not yet available as pre-built ``wheel`` files may also help with obtaining other binary extensions without needing to build them locally. .. seealso:: `Python Packaging User Guide: Binary Extensions `__ PK 3]6ȥusing/unix.rst.txtnu[.. highlightlang:: sh .. _using-on-unix: ******************************** Using Python on Unix platforms ******************************** .. sectionauthor:: Shriphani Palakodety Getting and installing the latest version of Python =================================================== On Linux -------- Python comes preinstalled on most Linux distributions, and is available as a package on all others. However there are certain features you might want to use that are not available on your distro's package. You can easily compile the latest version of Python from source. In the event that Python doesn't come preinstalled and isn't in the repositories as well, you can easily make packages for your own distro. Have a look at the following links: .. seealso:: https://www.debian.org/doc/manuals/maint-guide/first.en.html for Debian users https://en.opensuse.org/Portal:Packaging for OpenSuse users https://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/RPM_Guide/ch-creating-rpms.html for Fedora users http://www.slackbook.org/html/package-management-making-packages.html for Slackware users On FreeBSD and OpenBSD ---------------------- * FreeBSD users, to add the package use:: pkg install python3 * OpenBSD users, to add the package use:: pkg_add -r python pkg_add ftp://ftp.openbsd.org/pub/OpenBSD/4.2/packages//python-.tgz For example i386 users get the 2.5.1 version of Python using:: pkg_add ftp://ftp.openbsd.org/pub/OpenBSD/4.2/packages/i386/python-2.5.1p2.tgz On OpenSolaris -------------- You can get Python from `OpenCSW `_. Various versions of Python are available and can be installed with e.g. ``pkgutil -i python27``. .. _building-python-on-unix: Building Python =============== If you want to compile CPython yourself, first thing you should do is get the `source `_. You can download either the latest release's source or just grab a fresh `clone `_. (If you want to contribute patches, you will need a clone.) The build process consists in the usual :: ./configure make make install invocations. Configuration options and caveats for specific Unix platforms are extensively documented in the :source:`README.rst` file in the root of the Python source tree. .. warning:: ``make install`` can overwrite or masquerade the :file:`python3` binary. ``make altinstall`` is therefore recommended instead of ``make install`` since it only installs :file:`{exec_prefix}/bin/python{version}`. Python-related paths and files ============================== These are subject to difference depending on local installation conventions; :envvar:`prefix` (``${prefix}``) and :envvar:`exec_prefix` (``${exec_prefix}``) are installation-dependent and should be interpreted as for GNU software; they may be the same. For example, on most Linux systems, the default for both is :file:`/usr`. +-----------------------------------------------+------------------------------------------+ | File/directory | Meaning | +===============================================+==========================================+ | :file:`{exec_prefix}/bin/python3` | Recommended location of the interpreter. | +-----------------------------------------------+------------------------------------------+ | :file:`{prefix}/lib/python{version}`, | Recommended locations of the directories | | :file:`{exec_prefix}/lib/python{version}` | containing the standard modules. | +-----------------------------------------------+------------------------------------------+ | :file:`{prefix}/include/python{version}`, | Recommended locations of the directories | | :file:`{exec_prefix}/include/python{version}` | containing the include files needed for | | | developing Python extensions and | | | embedding the interpreter. | +-----------------------------------------------+------------------------------------------+ Miscellaneous ============= To easily use Python scripts on Unix, you need to make them executable, e.g. with .. code-block:: shell-session $ chmod +x script and put an appropriate Shebang line at the top of the script. A good choice is usually :: #!/usr/bin/env python3 which searches for the Python interpreter in the whole :envvar:`PATH`. However, some Unices may not have the :program:`env` command, so you may need to hardcode ``/usr/bin/python3`` as the interpreter path. To use shell commands in your Python scripts, look at the :mod:`subprocess` module. Editors and IDEs ================ There are a number of IDEs that support Python programming language. Many editors and IDEs provide syntax highlighting, debugging tools, and PEP-8 checks. Please go to `Python Editors `_ and `Integrated Development Environments `_ for a comprehensive list. PK 3]dCddusing/cmdline.rst.txtnu[.. highlightlang:: sh .. ATTENTION: You probably should update Misc/python.man, too, if you modify this file. .. _using-on-general: Command line and environment ============================ The CPython interpreter scans the command line and the environment for various settings. .. impl-detail:: Other implementations' command line schemes may differ. See :ref:`implementations` for further resources. .. _using-on-cmdline: Command line ------------ When invoking Python, you may specify any of these options:: python [-bBdEhiIOqsSuvVWx?] [-c command | -m module-name | script | - ] [args] The most common use case is, of course, a simple invocation of a script:: python myscript.py .. _using-on-interface-options: Interface options ~~~~~~~~~~~~~~~~~ The interpreter interface resembles that of the UNIX shell, but provides some additional methods of invocation: * When called with standard input connected to a tty device, it prompts for commands and executes them until an EOF (an end-of-file character, you can produce that with :kbd:`Ctrl-D` on UNIX or :kbd:`Ctrl-Z, Enter` on Windows) is read. * When called with a file name argument or with a file as standard input, it reads and executes a script from that file. * When called with a directory name argument, it reads and executes an appropriately named script from that directory. * When called with ``-c command``, it executes the Python statement(s) given as *command*. Here *command* may contain multiple statements separated by newlines. Leading whitespace is significant in Python statements! * When called with ``-m module-name``, the given module is located on the Python module path and executed as a script. In non-interactive mode, the entire input is parsed before it is executed. An interface option terminates the list of options consumed by the interpreter, all consecutive arguments will end up in :data:`sys.argv` -- note that the first element, subscript zero (``sys.argv[0]``), is a string reflecting the program's source. .. cmdoption:: -c Execute the Python code in *command*. *command* can be one or more statements separated by newlines, with significant leading whitespace as in normal module code. If this option is given, the first element of :data:`sys.argv` will be ``"-c"`` and the current directory will be added to the start of :data:`sys.path` (allowing modules in that directory to be imported as top level modules). .. cmdoption:: -m Search :data:`sys.path` for the named module and execute its contents as the :mod:`__main__` module. Since the argument is a *module* name, you must not give a file extension (``.py``). The module name should be a valid absolute Python module name, but the implementation may not always enforce this (e.g. it may allow you to use a name that includes a hyphen). Package names (including namespace packages) are also permitted. When a package name is supplied instead of a normal module, the interpreter will execute ``.__main__`` as the main module. This behaviour is deliberately similar to the handling of directories and zipfiles that are passed to the interpreter as the script argument. .. note:: This option cannot be used with built-in modules and extension modules written in C, since they do not have Python module files. However, it can still be used for precompiled modules, even if the original source file is not available. If this option is given, the first element of :data:`sys.argv` will be the full path to the module file (while the module file is being located, the first element will be set to ``"-m"``). As with the :option:`-c` option, the current directory will be added to the start of :data:`sys.path`. Many standard library modules contain code that is invoked on their execution as a script. An example is the :mod:`timeit` module:: python -mtimeit -s 'setup here' 'benchmarked code here' python -mtimeit -h # for details .. seealso:: :func:`runpy.run_module` Equivalent functionality directly available to Python code :pep:`338` -- Executing modules as scripts .. versionchanged:: 3.1 Supply the package name to run a ``__main__`` submodule. .. versionchanged:: 3.4 namespace packages are also supported .. describe:: - Read commands from standard input (:data:`sys.stdin`). If standard input is a terminal, :option:`-i` is implied. If this option is given, the first element of :data:`sys.argv` will be ``"-"`` and the current directory will be added to the start of :data:`sys.path`. .. describe:: `` and ````). .. method:: HTMLParser.handle_entityref(name) This method is called to process a named character reference of the form ``&name;`` (e.g. ``>``), where *name* is a general entity reference (e.g. ``'gt'``). This method is never called if *convert_charrefs* is ``True``. .. method:: HTMLParser.handle_charref(name) This method is called to process decimal and hexadecimal numeric character references of the form ``&#NNN;`` and ``&#xNNN;``. For example, the decimal equivalent for ``>`` is ``>``, whereas the hexadecimal is ``>``; in this case the method will receive ``'62'`` or ``'x3E'``. This method is never called if *convert_charrefs* is ``True``. .. method:: HTMLParser.handle_comment(data) This method is called when a comment is encountered (e.g. ````). For example, the comment ```` will cause this method to be called with the argument ``' comment '``. The content of Internet Explorer conditional comments (condcoms) will also be sent to this method, so, for ````, this method will receive ``'[if IE 9]>IE9-specific content``). The *decl* parameter will be the entire contents of the declaration inside the ```` markup (e.g. ``'DOCTYPE html'``). .. method:: HTMLParser.handle_pi(data) Method called when a processing instruction is encountered. The *data* parameter will contain the entire processing instruction. For example, for the processing instruction ````, this method would be called as ``handle_pi("proc color='red'")``. It is intended to be overridden by a derived class; the base class implementation does nothing. .. note:: The :class:`HTMLParser` class uses the SGML syntactic rules for processing instructions. An XHTML processing instruction using the trailing ``'?'`` will cause the ``'?'`` to be included in *data*. .. method:: HTMLParser.unknown_decl(data) This method is called when an unrecognized declaration is read by the parser. The *data* parameter will be the entire contents of the declaration inside the ```` markup. It is sometimes useful to be overridden by a derived class. The base class implementation does nothing. .. _htmlparser-examples: Examples -------- The following class implements a parser that will be used to illustrate more examples:: from html.parser import HTMLParser from html.entities import name2codepoint class MyHTMLParser(HTMLParser): def handle_starttag(self, tag, attrs): print("Start tag:", tag) for attr in attrs: print(" attr:", attr) def handle_endtag(self, tag): print("End tag :", tag) def handle_data(self, data): print("Data :", data) def handle_comment(self, data): print("Comment :", data) def handle_entityref(self, name): c = chr(name2codepoint[name]) print("Named ent:", c) def handle_charref(self, name): if name.startswith('x'): c = chr(int(name[1:], 16)) else: c = chr(int(name)) print("Num ent :", c) def handle_decl(self, data): print("Decl :", data) parser = MyHTMLParser() Parsing a doctype:: >>> parser.feed('') Decl : DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd" Parsing an element with a few attributes and a title:: >>> parser.feed('The Python logo') Start tag: img attr: ('src', 'python-logo.png') attr: ('alt', 'The Python logo') >>> >>> parser.feed('

Python

') Start tag: h1 Data : Python End tag : h1 The content of ``script`` and ``style`` elements is returned as is, without further parsing:: >>> parser.feed('') Start tag: style attr: ('type', 'text/css') Data : #python { color: green } End tag : style >>> parser.feed('') Start tag: script attr: ('type', 'text/javascript') Data : alert("hello!"); End tag : script Parsing comments:: >>> parser.feed('' ... '') Comment : a comment Comment : [if IE 9]>IE-specific content'``):: >>> parser.feed('>>>') Named ent: > Num ent : > Num ent : > Feeding incomplete chunks to :meth:`~HTMLParser.feed` works, but :meth:`~HTMLParser.handle_data` might be called more than once (unless *convert_charrefs* is set to ``True``):: >>> for chunk in ['buff', 'ered ', 'text']: ... parser.feed(chunk) ... Start tag: span Data : buff Data : ered Data : text End tag : span Parsing invalid HTML (e.g. unquoted attributes) also works:: >>> parser.feed('

tag soup

') Start tag: p Start tag: a attr: ('class', 'link') attr: ('href', '#main') Data : tag soup End tag : p End tag : a PK 3]W$Vlibrary/stringprep.rst.txtnu[:mod:`stringprep` --- Internet String Preparation ================================================= .. module:: stringprep :synopsis: String preparation, as per RFC 3453 .. moduleauthor:: Martin v. Löwis .. sectionauthor:: Martin v. Löwis **Source code:** :source:`Lib/stringprep.py` -------------- When identifying things (such as host names) in the internet, it is often necessary to compare such identifications for "equality". Exactly how this comparison is executed may depend on the application domain, e.g. whether it should be case-insensitive or not. It may be also necessary to restrict the possible identifications, to allow only identifications consisting of "printable" characters. :rfc:`3454` defines a procedure for "preparing" Unicode strings in internet protocols. Before passing strings onto the wire, they are processed with the preparation procedure, after which they have a certain normalized form. The RFC defines a set of tables, which can be combined into profiles. Each profile must define which tables it uses, and what other optional parts of the ``stringprep`` procedure are part of the profile. One example of a ``stringprep`` profile is ``nameprep``, which is used for internationalized domain names. The module :mod:`stringprep` only exposes the tables from :rfc:`3454`. As these tables would be very large to represent them as dictionaries or lists, the module uses the Unicode character database internally. The module source code itself was generated using the ``mkstringprep.py`` utility. As a result, these tables are exposed as functions, not as data structures. There are two kinds of tables in the RFC: sets and mappings. For a set, :mod:`stringprep` provides the "characteristic function", i.e. a function that returns true if the parameter is part of the set. For mappings, it provides the mapping function: given the key, it returns the associated value. Below is a list of all functions available in the module. .. function:: in_table_a1(code) Determine whether *code* is in tableA.1 (Unassigned code points in Unicode 3.2). .. function:: in_table_b1(code) Determine whether *code* is in tableB.1 (Commonly mapped to nothing). .. function:: map_table_b2(code) Return the mapped value for *code* according to tableB.2 (Mapping for case-folding used with NFKC). .. function:: map_table_b3(code) Return the mapped value for *code* according to tableB.3 (Mapping for case-folding used with no normalization). .. function:: in_table_c11(code) Determine whether *code* is in tableC.1.1 (ASCII space characters). .. function:: in_table_c12(code) Determine whether *code* is in tableC.1.2 (Non-ASCII space characters). .. function:: in_table_c11_c12(code) Determine whether *code* is in tableC.1 (Space characters, union of C.1.1 and C.1.2). .. function:: in_table_c21(code) Determine whether *code* is in tableC.2.1 (ASCII control characters). .. function:: in_table_c22(code) Determine whether *code* is in tableC.2.2 (Non-ASCII control characters). .. function:: in_table_c21_c22(code) Determine whether *code* is in tableC.2 (Control characters, union of C.2.1 and C.2.2). .. function:: in_table_c3(code) Determine whether *code* is in tableC.3 (Private use). .. function:: in_table_c4(code) Determine whether *code* is in tableC.4 (Non-character code points). .. function:: in_table_c5(code) Determine whether *code* is in tableC.5 (Surrogate codes). .. function:: in_table_c6(code) Determine whether *code* is in tableC.6 (Inappropriate for plain text). .. function:: in_table_c7(code) Determine whether *code* is in tableC.7 (Inappropriate for canonical representation). .. function:: in_table_c8(code) Determine whether *code* is in tableC.8 (Change display properties or are deprecated). .. function:: in_table_c9(code) Determine whether *code* is in tableC.9 (Tagging characters). .. function:: in_table_d1(code) Determine whether *code* is in tableD.1 (Characters with bidirectional property "R" or "AL"). .. function:: in_table_d2(code) Determine whether *code* is in tableD.2 (Characters with bidirectional property "L"). PK 3]87&7&library/webbrowser.rst.txtnu[:mod:`webbrowser` --- Convenient Web-browser controller ======================================================= .. module:: webbrowser :synopsis: Easy-to-use controller for Web browsers. .. moduleauthor:: Fred L. Drake, Jr. .. sectionauthor:: Fred L. Drake, Jr. **Source code:** :source:`Lib/webbrowser.py` -------------- The :mod:`webbrowser` module provides a high-level interface to allow displaying Web-based documents to users. Under most circumstances, simply calling the :func:`.open` function from this module will do the right thing. Under Unix, graphical browsers are preferred under X11, but text-mode browsers will be used if graphical browsers are not available or an X11 display isn't available. If text-mode browsers are used, the calling process will block until the user exits the browser. If the environment variable :envvar:`BROWSER` exists, it is interpreted as the :data:`os.pathsep`-separated list of browsers to try ahead of the platform defaults. When the value of a list part contains the string ``%s``, then it is interpreted as a literal browser command line to be used with the argument URL substituted for ``%s``; if the part does not contain ``%s``, it is simply interpreted as the name of the browser to launch. [1]_ For non-Unix platforms, or when a remote browser is available on Unix, the controlling process will not wait for the user to finish with the browser, but allow the remote browser to maintain its own windows on the display. If remote browsers are not available on Unix, the controlling process will launch a new browser and wait. The script :program:`webbrowser` can be used as a command-line interface for the module. It accepts a URL as the argument. It accepts the following optional parameters: ``-n`` opens the URL in a new browser window, if possible; ``-t`` opens the URL in a new browser page ("tab"). The options are, naturally, mutually exclusive. Usage example:: python -m webbrowser -t "http://www.python.org" The following exception is defined: .. exception:: Error Exception raised when a browser control error occurs. The following functions are defined: .. function:: open(url, new=0, autoraise=True) Display *url* using the default browser. If *new* is 0, the *url* is opened in the same browser window if possible. If *new* is 1, a new browser window is opened if possible. If *new* is 2, a new browser page ("tab") is opened if possible. If *autoraise* is ``True``, the window is raised if possible (note that under many window managers this will occur regardless of the setting of this variable). Note that on some platforms, trying to open a filename using this function, may work and start the operating system's associated program. However, this is neither supported nor portable. .. function:: open_new(url) Open *url* in a new window of the default browser, if possible, otherwise, open *url* in the only browser window. .. function:: open_new_tab(url) Open *url* in a new page ("tab") of the default browser, if possible, otherwise equivalent to :func:`open_new`. .. function:: get(using=None) Return a controller object for the browser type *using*. If *using* is ``None``, return a controller for a default browser appropriate to the caller's environment. .. function:: register(name, constructor, instance=None) Register the browser type *name*. Once a browser type is registered, the :func:`get` function can return a controller for that browser type. If *instance* is not provided, or is ``None``, *constructor* will be called without parameters to create an instance when needed. If *instance* is provided, *constructor* will never be called, and may be ``None``. This entry point is only useful if you plan to either set the :envvar:`BROWSER` variable or call :func:`get` with a nonempty argument matching the name of a handler you declare. A number of browser types are predefined. This table gives the type names that may be passed to the :func:`get` function and the corresponding instantiations for the controller classes, all defined in this module. +------------------------+-----------------------------------------+-------+ | Type Name | Class Name | Notes | +========================+=========================================+=======+ | ``'mozilla'`` | :class:`Mozilla('mozilla')` | | +------------------------+-----------------------------------------+-------+ | ``'firefox'`` | :class:`Mozilla('mozilla')` | | +------------------------+-----------------------------------------+-------+ | ``'netscape'`` | :class:`Mozilla('netscape')` | | +------------------------+-----------------------------------------+-------+ | ``'galeon'`` | :class:`Galeon('galeon')` | | +------------------------+-----------------------------------------+-------+ | ``'epiphany'`` | :class:`Galeon('epiphany')` | | +------------------------+-----------------------------------------+-------+ | ``'skipstone'`` | :class:`BackgroundBrowser('skipstone')` | | +------------------------+-----------------------------------------+-------+ | ``'kfmclient'`` | :class:`Konqueror()` | \(1) | +------------------------+-----------------------------------------+-------+ | ``'konqueror'`` | :class:`Konqueror()` | \(1) | +------------------------+-----------------------------------------+-------+ | ``'kfm'`` | :class:`Konqueror()` | \(1) | +------------------------+-----------------------------------------+-------+ | ``'mosaic'`` | :class:`BackgroundBrowser('mosaic')` | | +------------------------+-----------------------------------------+-------+ | ``'opera'`` | :class:`Opera()` | | +------------------------+-----------------------------------------+-------+ | ``'grail'`` | :class:`Grail()` | | +------------------------+-----------------------------------------+-------+ | ``'links'`` | :class:`GenericBrowser('links')` | | +------------------------+-----------------------------------------+-------+ | ``'elinks'`` | :class:`Elinks('elinks')` | | +------------------------+-----------------------------------------+-------+ | ``'lynx'`` | :class:`GenericBrowser('lynx')` | | +------------------------+-----------------------------------------+-------+ | ``'w3m'`` | :class:`GenericBrowser('w3m')` | | +------------------------+-----------------------------------------+-------+ | ``'windows-default'`` | :class:`WindowsDefault` | \(2) | +------------------------+-----------------------------------------+-------+ | ``'macosx'`` | :class:`MacOSX('default')` | \(3) | +------------------------+-----------------------------------------+-------+ | ``'safari'`` | :class:`MacOSX('safari')` | \(3) | +------------------------+-----------------------------------------+-------+ | ``'google-chrome'`` | :class:`Chrome('google-chrome')` | | +------------------------+-----------------------------------------+-------+ | ``'chrome'`` | :class:`Chrome('chrome')` | | +------------------------+-----------------------------------------+-------+ | ``'chromium'`` | :class:`Chromium('chromium')` | | +------------------------+-----------------------------------------+-------+ | ``'chromium-browser'`` | :class:`Chromium('chromium-browser')` | | +------------------------+-----------------------------------------+-------+ Notes: (1) "Konqueror" is the file manager for the KDE desktop environment for Unix, and only makes sense to use if KDE is running. Some way of reliably detecting KDE would be nice; the :envvar:`KDEDIR` variable is not sufficient. Note also that the name "kfm" is used even when using the :program:`konqueror` command with KDE 2 --- the implementation selects the best strategy for running Konqueror. (2) Only on Windows platforms. (3) Only on Mac OS X platform. .. versionadded:: 3.3 Support for Chrome/Chromium has been added. Here are some simple examples:: url = 'http://docs.python.org/' # Open URL in a new tab, if a browser window is already open. webbrowser.open_new_tab(url) # Open URL in new window, raising the window if possible. webbrowser.open_new(url) .. _browser-controllers: Browser Controller Objects -------------------------- Browser controllers provide these methods which parallel three of the module-level convenience functions: .. method:: controller.open(url, new=0, autoraise=True) Display *url* using the browser handled by this controller. If *new* is 1, a new browser window is opened if possible. If *new* is 2, a new browser page ("tab") is opened if possible. .. method:: controller.open_new(url) Open *url* in a new window of the browser handled by this controller, if possible, otherwise, open *url* in the only browser window. Alias :func:`open_new`. .. method:: controller.open_new_tab(url) Open *url* in a new page ("tab") of the browser handled by this controller, if possible, otherwise equivalent to :func:`open_new`. .. rubric:: Footnotes .. [1] Executables named here without a full path will be searched in the directories given in the :envvar:`PATH` environment variable. PK 3] ??library/logging.rst.txtnu[:mod:`logging` --- Logging facility for Python ============================================== .. module:: logging :synopsis: Flexible event logging system for applications. .. moduleauthor:: Vinay Sajip .. sectionauthor:: Vinay Sajip **Source code:** :source:`Lib/logging/__init__.py` .. index:: pair: Errors; logging .. sidebar:: Important This page contains the API reference information. For tutorial information and discussion of more advanced topics, see * :ref:`Basic Tutorial ` * :ref:`Advanced Tutorial ` * :ref:`Logging Cookbook ` -------------- This module defines functions and classes which implement a flexible event logging system for applications and libraries. The key benefit of having the logging API provided by a standard library module is that all Python modules can participate in logging, so your application log can include your own messages integrated with messages from third-party modules. The module provides a lot of functionality and flexibility. If you are unfamiliar with logging, the best way to get to grips with it is to see the tutorials (see the links on the right). The basic classes defined by the module, together with their functions, are listed below. * Loggers expose the interface that application code directly uses. * Handlers send the log records (created by loggers) to the appropriate destination. * Filters provide a finer grained facility for determining which log records to output. * Formatters specify the layout of log records in the final output. .. _logger: Logger Objects -------------- Loggers have the following attributes and methods. Note that Loggers are never instantiated directly, but always through the module-level function ``logging.getLogger(name)``. Multiple calls to :func:`getLogger` with the same name will always return a reference to the same Logger object. The ``name`` is potentially a period-separated hierarchical value, like ``foo.bar.baz`` (though it could also be just plain ``foo``, for example). Loggers that are further down in the hierarchical list are children of loggers higher up in the list. For example, given a logger with a name of ``foo``, loggers with names of ``foo.bar``, ``foo.bar.baz``, and ``foo.bam`` are all descendants of ``foo``. The logger name hierarchy is analogous to the Python package hierarchy, and identical to it if you organise your loggers on a per-module basis using the recommended construction ``logging.getLogger(__name__)``. That's because in a module, ``__name__`` is the module's name in the Python package namespace. .. class:: Logger .. attribute:: Logger.propagate If this attribute evaluates to true, events logged to this logger will be passed to the handlers of higher level (ancestor) loggers, in addition to any handlers attached to this logger. Messages are passed directly to the ancestor loggers' handlers - neither the level nor filters of the ancestor loggers in question are considered. If this evaluates to false, logging messages are not passed to the handlers of ancestor loggers. The constructor sets this attribute to ``True``. .. note:: If you attach a handler to a logger *and* one or more of its ancestors, it may emit the same record multiple times. In general, you should not need to attach a handler to more than one logger - if you just attach it to the appropriate logger which is highest in the logger hierarchy, then it will see all events logged by all descendant loggers, provided that their propagate setting is left set to ``True``. A common scenario is to attach handlers only to the root logger, and to let propagation take care of the rest. .. method:: Logger.setLevel(level) Sets the threshold for this logger to *level*. Logging messages which are less severe than *level* will be ignored; logging messages which have severity *level* or higher will be emitted by whichever handler or handlers service this logger, unless a handler's level has been set to a higher severity level than *level*. When a logger is created, the level is set to :const:`NOTSET` (which causes all messages to be processed when the logger is the root logger, or delegation to the parent when the logger is a non-root logger). Note that the root logger is created with level :const:`WARNING`. The term 'delegation to the parent' means that if a logger has a level of NOTSET, its chain of ancestor loggers is traversed until either an ancestor with a level other than NOTSET is found, or the root is reached. If an ancestor is found with a level other than NOTSET, then that ancestor's level is treated as the effective level of the logger where the ancestor search began, and is used to determine how a logging event is handled. If the root is reached, and it has a level of NOTSET, then all messages will be processed. Otherwise, the root's level will be used as the effective level. See :ref:`levels` for a list of levels. .. versionchanged:: 3.2 The *level* parameter now accepts a string representation of the level such as 'INFO' as an alternative to the integer constants such as :const:`INFO`. Note, however, that levels are internally stored as integers, and methods such as e.g. :meth:`getEffectiveLevel` and :meth:`isEnabledFor` will return/expect to be passed integers. .. method:: Logger.isEnabledFor(lvl) Indicates if a message of severity *lvl* would be processed by this logger. This method checks first the module-level level set by ``logging.disable(lvl)`` and then the logger's effective level as determined by :meth:`getEffectiveLevel`. .. method:: Logger.getEffectiveLevel() Indicates the effective level for this logger. If a value other than :const:`NOTSET` has been set using :meth:`setLevel`, it is returned. Otherwise, the hierarchy is traversed towards the root until a value other than :const:`NOTSET` is found, and that value is returned. The value returned is an integer, typically one of :const:`logging.DEBUG`, :const:`logging.INFO` etc. .. method:: Logger.getChild(suffix) Returns a logger which is a descendant to this logger, as determined by the suffix. Thus, ``logging.getLogger('abc').getChild('def.ghi')`` would return the same logger as would be returned by ``logging.getLogger('abc.def.ghi')``. This is a convenience method, useful when the parent logger is named using e.g. ``__name__`` rather than a literal string. .. versionadded:: 3.2 .. method:: Logger.debug(msg, *args, **kwargs) Logs a message with level :const:`DEBUG` on this logger. The *msg* is the message format string, and the *args* are the arguments which are merged into *msg* using the string formatting operator. (Note that this means that you can use keywords in the format string, together with a single dictionary argument.) There are three keyword arguments in *kwargs* which are inspected: *exc_info*, *stack_info*, and *extra*. If *exc_info* does not evaluate as false, it causes exception information to be added to the logging message. If an exception tuple (in the format returned by :func:`sys.exc_info`) or an exception instance is provided, it is used; otherwise, :func:`sys.exc_info` is called to get the exception information. The second optional keyword argument is *stack_info*, which defaults to ``False``. If true, stack information is added to the logging message, including the actual logging call. Note that this is not the same stack information as that displayed through specifying *exc_info*: The former is stack frames from the bottom of the stack up to the logging call in the current thread, whereas the latter is information about stack frames which have been unwound, following an exception, while searching for exception handlers. You can specify *stack_info* independently of *exc_info*, e.g. to just show how you got to a certain point in your code, even when no exceptions were raised. The stack frames are printed following a header line which says: .. code-block:: none Stack (most recent call last): This mimics the ``Traceback (most recent call last):`` which is used when displaying exception frames. The third keyword argument is *extra* which can be used to pass a dictionary which is used to populate the __dict__ of the LogRecord created for the logging event with user-defined attributes. These custom attributes can then be used as you like. For example, they could be incorporated into logged messages. For example:: FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s' logging.basicConfig(format=FORMAT) d = {'clientip': '192.168.0.1', 'user': 'fbloggs'} logger = logging.getLogger('tcpserver') logger.warning('Protocol problem: %s', 'connection reset', extra=d) would print something like .. code-block:: none 2006-02-08 22:20:02,165 192.168.0.1 fbloggs Protocol problem: connection reset The keys in the dictionary passed in *extra* should not clash with the keys used by the logging system. (See the :class:`Formatter` documentation for more information on which keys are used by the logging system.) If you choose to use these attributes in logged messages, you need to exercise some care. In the above example, for instance, the :class:`Formatter` has been set up with a format string which expects 'clientip' and 'user' in the attribute dictionary of the LogRecord. If these are missing, the message will not be logged because a string formatting exception will occur. So in this case, you always need to pass the *extra* dictionary with these keys. While this might be annoying, this feature is intended for use in specialized circumstances, such as multi-threaded servers where the same code executes in many contexts, and interesting conditions which arise are dependent on this context (such as remote client IP address and authenticated user name, in the above example). In such circumstances, it is likely that specialized :class:`Formatter`\ s would be used with particular :class:`Handler`\ s. .. versionadded:: 3.2 The *stack_info* parameter was added. .. versionchanged:: 3.5 The *exc_info* parameter can now accept exception instances. .. method:: Logger.info(msg, *args, **kwargs) Logs a message with level :const:`INFO` on this logger. The arguments are interpreted as for :meth:`debug`. .. method:: Logger.warning(msg, *args, **kwargs) Logs a message with level :const:`WARNING` on this logger. The arguments are interpreted as for :meth:`debug`. .. note:: There is an obsolete method ``warn`` which is functionally identical to ``warning``. As ``warn`` is deprecated, please do not use it - use ``warning`` instead. .. method:: Logger.error(msg, *args, **kwargs) Logs a message with level :const:`ERROR` on this logger. The arguments are interpreted as for :meth:`debug`. .. method:: Logger.critical(msg, *args, **kwargs) Logs a message with level :const:`CRITICAL` on this logger. The arguments are interpreted as for :meth:`debug`. .. method:: Logger.log(lvl, msg, *args, **kwargs) Logs a message with integer level *lvl* on this logger. The other arguments are interpreted as for :meth:`debug`. .. method:: Logger.exception(msg, *args, **kwargs) Logs a message with level :const:`ERROR` on this logger. The arguments are interpreted as for :meth:`debug`. Exception info is added to the logging message. This method should only be called from an exception handler. .. method:: Logger.addFilter(filter) Adds the specified filter *filter* to this logger. .. method:: Logger.removeFilter(filter) Removes the specified filter *filter* from this logger. .. method:: Logger.filter(record) Applies this logger's filters to the record and returns a true value if the record is to be processed. The filters are consulted in turn, until one of them returns a false value. If none of them return a false value, the record will be processed (passed to handlers). If one returns a false value, no further processing of the record occurs. .. method:: Logger.addHandler(hdlr) Adds the specified handler *hdlr* to this logger. .. method:: Logger.removeHandler(hdlr) Removes the specified handler *hdlr* from this logger. .. method:: Logger.findCaller(stack_info=False) Finds the caller's source filename and line number. Returns the filename, line number, function name and stack information as a 4-element tuple. The stack information is returned as ``None`` unless *stack_info* is ``True``. .. method:: Logger.handle(record) Handles a record by passing it to all handlers associated with this logger and its ancestors (until a false value of *propagate* is found). This method is used for unpickled records received from a socket, as well as those created locally. Logger-level filtering is applied using :meth:`~Logger.filter`. .. method:: Logger.makeRecord(name, lvl, fn, lno, msg, args, exc_info, func=None, extra=None, sinfo=None) This is a factory method which can be overridden in subclasses to create specialized :class:`LogRecord` instances. .. method:: Logger.hasHandlers() Checks to see if this logger has any handlers configured. This is done by looking for handlers in this logger and its parents in the logger hierarchy. Returns ``True`` if a handler was found, else ``False``. The method stops searching up the hierarchy whenever a logger with the 'propagate' attribute set to false is found - that will be the last logger which is checked for the existence of handlers. .. versionadded:: 3.2 .. versionchanged:: 3.7 Loggers can now be picked and unpickled. .. _levels: Logging Levels -------------- The numeric values of logging levels are given in the following table. These are primarily of interest if you want to define your own levels, and need them to have specific values relative to the predefined levels. If you define a level with the same numeric value, it overwrites the predefined value; the predefined name is lost. +--------------+---------------+ | Level | Numeric value | +==============+===============+ | ``CRITICAL`` | 50 | +--------------+---------------+ | ``ERROR`` | 40 | +--------------+---------------+ | ``WARNING`` | 30 | +--------------+---------------+ | ``INFO`` | 20 | +--------------+---------------+ | ``DEBUG`` | 10 | +--------------+---------------+ | ``NOTSET`` | 0 | +--------------+---------------+ .. _handler: Handler Objects --------------- Handlers have the following attributes and methods. Note that :class:`Handler` is never instantiated directly; this class acts as a base for more useful subclasses. However, the :meth:`__init__` method in subclasses needs to call :meth:`Handler.__init__`. .. class:: Handler .. method:: Handler.__init__(level=NOTSET) Initializes the :class:`Handler` instance by setting its level, setting the list of filters to the empty list and creating a lock (using :meth:`createLock`) for serializing access to an I/O mechanism. .. method:: Handler.createLock() Initializes a thread lock which can be used to serialize access to underlying I/O functionality which may not be threadsafe. .. method:: Handler.acquire() Acquires the thread lock created with :meth:`createLock`. .. method:: Handler.release() Releases the thread lock acquired with :meth:`acquire`. .. method:: Handler.setLevel(level) Sets the threshold for this handler to *level*. Logging messages which are less severe than *level* will be ignored. When a handler is created, the level is set to :const:`NOTSET` (which causes all messages to be processed). See :ref:`levels` for a list of levels. .. versionchanged:: 3.2 The *level* parameter now accepts a string representation of the level such as 'INFO' as an alternative to the integer constants such as :const:`INFO`. .. method:: Handler.setFormatter(fmt) Sets the :class:`Formatter` for this handler to *fmt*. .. method:: Handler.addFilter(filter) Adds the specified filter *filter* to this handler. .. method:: Handler.removeFilter(filter) Removes the specified filter *filter* from this handler. .. method:: Handler.filter(record) Applies this handler's filters to the record and returns a true value if the record is to be processed. The filters are consulted in turn, until one of them returns a false value. If none of them return a false value, the record will be emitted. If one returns a false value, the handler will not emit the record. .. method:: Handler.flush() Ensure all logging output has been flushed. This version does nothing and is intended to be implemented by subclasses. .. method:: Handler.close() Tidy up any resources used by the handler. This version does no output but removes the handler from an internal list of handlers which is closed when :func:`shutdown` is called. Subclasses should ensure that this gets called from overridden :meth:`close` methods. .. method:: Handler.handle(record) Conditionally emits the specified logging record, depending on filters which may have been added to the handler. Wraps the actual emission of the record with acquisition/release of the I/O thread lock. .. method:: Handler.handleError(record) This method should be called from handlers when an exception is encountered during an :meth:`emit` call. If the module-level attribute ``raiseExceptions`` is ``False``, exceptions get silently ignored. This is what is mostly wanted for a logging system - most users will not care about errors in the logging system, they are more interested in application errors. You could, however, replace this with a custom handler if you wish. The specified record is the one which was being processed when the exception occurred. (The default value of ``raiseExceptions`` is ``True``, as that is more useful during development). .. method:: Handler.format(record) Do formatting for a record - if a formatter is set, use it. Otherwise, use the default formatter for the module. .. method:: Handler.emit(record) Do whatever it takes to actually log the specified logging record. This version is intended to be implemented by subclasses and so raises a :exc:`NotImplementedError`. For a list of handlers included as standard, see :mod:`logging.handlers`. .. _formatter-objects: Formatter Objects ----------------- .. currentmodule:: logging :class:`Formatter` objects have the following attributes and methods. They are responsible for converting a :class:`LogRecord` to (usually) a string which can be interpreted by either a human or an external system. The base :class:`Formatter` allows a formatting string to be specified. If none is supplied, the default value of ``'%(message)s'`` is used, which just includes the message in the logging call. To have additional items of information in the formatted output (such as a timestamp), keep reading. A Formatter can be initialized with a format string which makes use of knowledge of the :class:`LogRecord` attributes - such as the default value mentioned above making use of the fact that the user's message and arguments are pre-formatted into a :class:`LogRecord`'s *message* attribute. This format string contains standard Python %-style mapping keys. See section :ref:`old-string-formatting` for more information on string formatting. The useful mapping keys in a :class:`LogRecord` are given in the section on :ref:`logrecord-attributes`. .. class:: Formatter(fmt=None, datefmt=None, style='%') Returns a new instance of the :class:`Formatter` class. The instance is initialized with a format string for the message as a whole, as well as a format string for the date/time portion of a message. If no *fmt* is specified, ``'%(message)s'`` is used. If no *datefmt* is specified, a format is used which is described in the :meth:`formatTime` documentation. The *style* parameter can be one of '%', '{' or '$' and determines how the format string will be merged with its data: using one of %-formatting, :meth:`str.format` or :class:`string.Template`. See :ref:`formatting-styles` for more information on using {- and $-formatting for log messages. .. versionchanged:: 3.2 The *style* parameter was added. .. method:: format(record) The record's attribute dictionary is used as the operand to a string formatting operation. Returns the resulting string. Before formatting the dictionary, a couple of preparatory steps are carried out. The *message* attribute of the record is computed using *msg* % *args*. If the formatting string contains ``'(asctime)'``, :meth:`formatTime` is called to format the event time. If there is exception information, it is formatted using :meth:`formatException` and appended to the message. Note that the formatted exception information is cached in attribute *exc_text*. This is useful because the exception information can be pickled and sent across the wire, but you should be careful if you have more than one :class:`Formatter` subclass which customizes the formatting of exception information. In this case, you will have to clear the cached value after a formatter has done its formatting, so that the next formatter to handle the event doesn't use the cached value but recalculates it afresh. If stack information is available, it's appended after the exception information, using :meth:`formatStack` to transform it if necessary. .. method:: formatTime(record, datefmt=None) This method should be called from :meth:`format` by a formatter which wants to make use of a formatted time. This method can be overridden in formatters to provide for any specific requirement, but the basic behavior is as follows: if *datefmt* (a string) is specified, it is used with :func:`time.strftime` to format the creation time of the record. Otherwise, the format '%Y-%m-%d %H:%M:%S,uuu' is used, where the uuu part is a millisecond value and the other letters are as per the :func:`time.strftime` documentation. An example time in this format is ``2003-01-23 00:29:50,411``. The resulting string is returned. This function uses a user-configurable function to convert the creation time to a tuple. By default, :func:`time.localtime` is used; to change this for a particular formatter instance, set the ``converter`` attribute to a function with the same signature as :func:`time.localtime` or :func:`time.gmtime`. To change it for all formatters, for example if you want all logging times to be shown in GMT, set the ``converter`` attribute in the ``Formatter`` class. .. versionchanged:: 3.3 Previously, the default format was hard-coded as in this example: ``2010-09-06 22:38:15,292`` where the part before the comma is handled by a strptime format string (``'%Y-%m-%d %H:%M:%S'``), and the part after the comma is a millisecond value. Because strptime does not have a format placeholder for milliseconds, the millisecond value is appended using another format string, ``'%s,%03d'`` --- and both of these format strings have been hardcoded into this method. With the change, these strings are defined as class-level attributes which can be overridden at the instance level when desired. The names of the attributes are ``default_time_format`` (for the strptime format string) and ``default_msec_format`` (for appending the millisecond value). .. method:: formatException(exc_info) Formats the specified exception information (a standard exception tuple as returned by :func:`sys.exc_info`) as a string. This default implementation just uses :func:`traceback.print_exception`. The resulting string is returned. .. method:: formatStack(stack_info) Formats the specified stack information (a string as returned by :func:`traceback.print_stack`, but with the last newline removed) as a string. This default implementation just returns the input value. .. _filter: Filter Objects -------------- ``Filters`` can be used by ``Handlers`` and ``Loggers`` for more sophisticated filtering than is provided by levels. The base filter class only allows events which are below a certain point in the logger hierarchy. For example, a filter initialized with 'A.B' will allow events logged by loggers 'A.B', 'A.B.C', 'A.B.C.D', 'A.B.D' etc. but not 'A.BB', 'B.A.B' etc. If initialized with the empty string, all events are passed. .. class:: Filter(name='') Returns an instance of the :class:`Filter` class. If *name* is specified, it names a logger which, together with its children, will have its events allowed through the filter. If *name* is the empty string, allows every event. .. method:: filter(record) Is the specified record to be logged? Returns zero for no, nonzero for yes. If deemed appropriate, the record may be modified in-place by this method. Note that filters attached to handlers are consulted before an event is emitted by the handler, whereas filters attached to loggers are consulted whenever an event is logged (using :meth:`debug`, :meth:`info`, etc.), before sending an event to handlers. This means that events which have been generated by descendant loggers will not be filtered by a logger's filter setting, unless the filter has also been applied to those descendant loggers. You don't actually need to subclass ``Filter``: you can pass any instance which has a ``filter`` method with the same semantics. .. versionchanged:: 3.2 You don't need to create specialized ``Filter`` classes, or use other classes with a ``filter`` method: you can use a function (or other callable) as a filter. The filtering logic will check to see if the filter object has a ``filter`` attribute: if it does, it's assumed to be a ``Filter`` and its :meth:`~Filter.filter` method is called. Otherwise, it's assumed to be a callable and called with the record as the single parameter. The returned value should conform to that returned by :meth:`~Filter.filter`. Although filters are used primarily to filter records based on more sophisticated criteria than levels, they get to see every record which is processed by the handler or logger they're attached to: this can be useful if you want to do things like counting how many records were processed by a particular logger or handler, or adding, changing or removing attributes in the LogRecord being processed. Obviously changing the LogRecord needs to be done with some care, but it does allow the injection of contextual information into logs (see :ref:`filters-contextual`). .. _log-record: LogRecord Objects ----------------- :class:`LogRecord` instances are created automatically by the :class:`Logger` every time something is logged, and can be created manually via :func:`makeLogRecord` (for example, from a pickled event received over the wire). .. class:: LogRecord(name, level, pathname, lineno, msg, args, exc_info, func=None, sinfo=None) Contains all the information pertinent to the event being logged. The primary information is passed in :attr:`msg` and :attr:`args`, which are combined using ``msg % args`` to create the :attr:`message` field of the record. :param name: The name of the logger used to log the event represented by this LogRecord. Note that this name will always have this value, even though it may be emitted by a handler attached to a different (ancestor) logger. :param level: The numeric level of the logging event (one of DEBUG, INFO etc.) Note that this is converted to *two* attributes of the LogRecord: ``levelno`` for the numeric value and ``levelname`` for the corresponding level name. :param pathname: The full pathname of the source file where the logging call was made. :param lineno: The line number in the source file where the logging call was made. :param msg: The event description message, possibly a format string with placeholders for variable data. :param args: Variable data to merge into the *msg* argument to obtain the event description. :param exc_info: An exception tuple with the current exception information, or ``None`` if no exception information is available. :param func: The name of the function or method from which the logging call was invoked. :param sinfo: A text string representing stack information from the base of the stack in the current thread, up to the logging call. .. method:: getMessage() Returns the message for this :class:`LogRecord` instance after merging any user-supplied arguments with the message. If the user-supplied message argument to the logging call is not a string, :func:`str` is called on it to convert it to a string. This allows use of user-defined classes as messages, whose ``__str__`` method can return the actual format string to be used. .. versionchanged:: 3.2 The creation of a ``LogRecord`` has been made more configurable by providing a factory which is used to create the record. The factory can be set using :func:`getLogRecordFactory` and :func:`setLogRecordFactory` (see this for the factory's signature). This functionality can be used to inject your own values into a LogRecord at creation time. You can use the following pattern:: old_factory = logging.getLogRecordFactory() def record_factory(*args, **kwargs): record = old_factory(*args, **kwargs) record.custom_attribute = 0xdecafbad return record logging.setLogRecordFactory(record_factory) With this pattern, multiple factories could be chained, and as long as they don't overwrite each other's attributes or unintentionally overwrite the standard attributes listed above, there should be no surprises. .. _logrecord-attributes: LogRecord attributes -------------------- The LogRecord has a number of attributes, most of which are derived from the parameters to the constructor. (Note that the names do not always correspond exactly between the LogRecord constructor parameters and the LogRecord attributes.) These attributes can be used to merge data from the record into the format string. The following table lists (in alphabetical order) the attribute names, their meanings and the corresponding placeholder in a %-style format string. If you are using {}-formatting (:func:`str.format`), you can use ``{attrname}`` as the placeholder in the format string. If you are using $-formatting (:class:`string.Template`), use the form ``${attrname}``. In both cases, of course, replace ``attrname`` with the actual attribute name you want to use. In the case of {}-formatting, you can specify formatting flags by placing them after the attribute name, separated from it with a colon. For example: a placeholder of ``{msecs:03d}`` would format a millisecond value of ``4`` as ``004``. Refer to the :meth:`str.format` documentation for full details on the options available to you. +----------------+-------------------------+-----------------------------------------------+ | Attribute name | Format | Description | +================+=========================+===============================================+ | args | You shouldn't need to | The tuple of arguments merged into ``msg`` to | | | format this yourself. | produce ``message``, or a dict whose values | | | | are used for the merge (when there is only one| | | | argument, and it is a dictionary). | +----------------+-------------------------+-----------------------------------------------+ | asctime | ``%(asctime)s`` | Human-readable time when the | | | | :class:`LogRecord` was created. By default | | | | this is of the form '2003-07-08 16:49:45,896' | | | | (the numbers after the comma are millisecond | | | | portion of the time). | +----------------+-------------------------+-----------------------------------------------+ | created | ``%(created)f`` | Time when the :class:`LogRecord` was created | | | | (as returned by :func:`time.time`). | +----------------+-------------------------+-----------------------------------------------+ | exc_info | You shouldn't need to | Exception tuple (à la ``sys.exc_info``) or, | | | format this yourself. | if no exception has occurred, ``None``. | +----------------+-------------------------+-----------------------------------------------+ | filename | ``%(filename)s`` | Filename portion of ``pathname``. | +----------------+-------------------------+-----------------------------------------------+ | funcName | ``%(funcName)s`` | Name of function containing the logging call. | +----------------+-------------------------+-----------------------------------------------+ | levelname | ``%(levelname)s`` | Text logging level for the message | | | | (``'DEBUG'``, ``'INFO'``, ``'WARNING'``, | | | | ``'ERROR'``, ``'CRITICAL'``). | +----------------+-------------------------+-----------------------------------------------+ | levelno | ``%(levelno)s`` | Numeric logging level for the message | | | | (:const:`DEBUG`, :const:`INFO`, | | | | :const:`WARNING`, :const:`ERROR`, | | | | :const:`CRITICAL`). | +----------------+-------------------------+-----------------------------------------------+ | lineno | ``%(lineno)d`` | Source line number where the logging call was | | | | issued (if available). | +----------------+-------------------------+-----------------------------------------------+ | message | ``%(message)s`` | The logged message, computed as ``msg % | | | | args``. This is set when | | | | :meth:`Formatter.format` is invoked. | +----------------+-------------------------+-----------------------------------------------+ | module | ``%(module)s`` | Module (name portion of ``filename``). | +----------------+-------------------------+-----------------------------------------------+ | msecs | ``%(msecs)d`` | Millisecond portion of the time when the | | | | :class:`LogRecord` was created. | +----------------+-------------------------+-----------------------------------------------+ | msg | You shouldn't need to | The format string passed in the original | | | format this yourself. | logging call. Merged with ``args`` to | | | | produce ``message``, or an arbitrary object | | | | (see :ref:`arbitrary-object-messages`). | +----------------+-------------------------+-----------------------------------------------+ | name | ``%(name)s`` | Name of the logger used to log the call. | +----------------+-------------------------+-----------------------------------------------+ | pathname | ``%(pathname)s`` | Full pathname of the source file where the | | | | logging call was issued (if available). | +----------------+-------------------------+-----------------------------------------------+ | process | ``%(process)d`` | Process ID (if available). | +----------------+-------------------------+-----------------------------------------------+ | processName | ``%(processName)s`` | Process name (if available). | +----------------+-------------------------+-----------------------------------------------+ | relativeCreated| ``%(relativeCreated)d`` | Time in milliseconds when the LogRecord was | | | | created, relative to the time the logging | | | | module was loaded. | +----------------+-------------------------+-----------------------------------------------+ | stack_info | You shouldn't need to | Stack frame information (where available) | | | format this yourself. | from the bottom of the stack in the current | | | | thread, up to and including the stack frame | | | | of the logging call which resulted in the | | | | creation of this record. | +----------------+-------------------------+-----------------------------------------------+ | thread | ``%(thread)d`` | Thread ID (if available). | +----------------+-------------------------+-----------------------------------------------+ | threadName | ``%(threadName)s`` | Thread name (if available). | +----------------+-------------------------+-----------------------------------------------+ .. versionchanged:: 3.1 *processName* was added. .. _logger-adapter: LoggerAdapter Objects --------------------- :class:`LoggerAdapter` instances are used to conveniently pass contextual information into logging calls. For a usage example, see the section on :ref:`adding contextual information to your logging output `. .. class:: LoggerAdapter(logger, extra) Returns an instance of :class:`LoggerAdapter` initialized with an underlying :class:`Logger` instance and a dict-like object. .. method:: process(msg, kwargs) Modifies the message and/or keyword arguments passed to a logging call in order to insert contextual information. This implementation takes the object passed as *extra* to the constructor and adds it to *kwargs* using key 'extra'. The return value is a (*msg*, *kwargs*) tuple which has the (possibly modified) versions of the arguments passed in. In addition to the above, :class:`LoggerAdapter` supports the following methods of :class:`Logger`: :meth:`~Logger.debug`, :meth:`~Logger.info`, :meth:`~Logger.warning`, :meth:`~Logger.error`, :meth:`~Logger.exception`, :meth:`~Logger.critical`, :meth:`~Logger.log`, :meth:`~Logger.isEnabledFor`, :meth:`~Logger.getEffectiveLevel`, :meth:`~Logger.setLevel` and :meth:`~Logger.hasHandlers`. These methods have the same signatures as their counterparts in :class:`Logger`, so you can use the two types of instances interchangeably. .. versionchanged:: 3.2 The :meth:`~Logger.isEnabledFor`, :meth:`~Logger.getEffectiveLevel`, :meth:`~Logger.setLevel` and :meth:`~Logger.hasHandlers` methods were added to :class:`LoggerAdapter`. These methods delegate to the underlying logger. Thread Safety ------------- The logging module is intended to be thread-safe without any special work needing to be done by its clients. It achieves this though using threading locks; there is one lock to serialize access to the module's shared data, and each handler also creates a lock to serialize access to its underlying I/O. If you are implementing asynchronous signal handlers using the :mod:`signal` module, you may not be able to use logging from within such handlers. This is because lock implementations in the :mod:`threading` module are not always re-entrant, and so cannot be invoked from such signal handlers. Module-Level Functions ---------------------- In addition to the classes described above, there are a number of module- level functions. .. function:: getLogger(name=None) Return a logger with the specified name or, if name is ``None``, return a logger which is the root logger of the hierarchy. If specified, the name is typically a dot-separated hierarchical name like *'a'*, *'a.b'* or *'a.b.c.d'*. Choice of these names is entirely up to the developer who is using logging. All calls to this function with a given name return the same logger instance. This means that logger instances never need to be passed between different parts of an application. .. function:: getLoggerClass() Return either the standard :class:`Logger` class, or the last class passed to :func:`setLoggerClass`. This function may be called from within a new class definition, to ensure that installing a customized :class:`Logger` class will not undo customizations already applied by other code. For example:: class MyLogger(logging.getLoggerClass()): # ... override behaviour here .. function:: getLogRecordFactory() Return a callable which is used to create a :class:`LogRecord`. .. versionadded:: 3.2 This function has been provided, along with :func:`setLogRecordFactory`, to allow developers more control over how the :class:`LogRecord` representing a logging event is constructed. See :func:`setLogRecordFactory` for more information about the how the factory is called. .. function:: debug(msg, *args, **kwargs) Logs a message with level :const:`DEBUG` on the root logger. The *msg* is the message format string, and the *args* are the arguments which are merged into *msg* using the string formatting operator. (Note that this means that you can use keywords in the format string, together with a single dictionary argument.) There are three keyword arguments in *kwargs* which are inspected: *exc_info* which, if it does not evaluate as false, causes exception information to be added to the logging message. If an exception tuple (in the format returned by :func:`sys.exc_info`) is provided, it is used; otherwise, :func:`sys.exc_info` is called to get the exception information. The second optional keyword argument is *stack_info*, which defaults to ``False``. If true, stack information is added to the logging message, including the actual logging call. Note that this is not the same stack information as that displayed through specifying *exc_info*: The former is stack frames from the bottom of the stack up to the logging call in the current thread, whereas the latter is information about stack frames which have been unwound, following an exception, while searching for exception handlers. You can specify *stack_info* independently of *exc_info*, e.g. to just show how you got to a certain point in your code, even when no exceptions were raised. The stack frames are printed following a header line which says: .. code-block:: none Stack (most recent call last): This mimics the ``Traceback (most recent call last):`` which is used when displaying exception frames. The third optional keyword argument is *extra* which can be used to pass a dictionary which is used to populate the __dict__ of the LogRecord created for the logging event with user-defined attributes. These custom attributes can then be used as you like. For example, they could be incorporated into logged messages. For example:: FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s' logging.basicConfig(format=FORMAT) d = {'clientip': '192.168.0.1', 'user': 'fbloggs'} logging.warning('Protocol problem: %s', 'connection reset', extra=d) would print something like: .. code-block:: none 2006-02-08 22:20:02,165 192.168.0.1 fbloggs Protocol problem: connection reset The keys in the dictionary passed in *extra* should not clash with the keys used by the logging system. (See the :class:`Formatter` documentation for more information on which keys are used by the logging system.) If you choose to use these attributes in logged messages, you need to exercise some care. In the above example, for instance, the :class:`Formatter` has been set up with a format string which expects 'clientip' and 'user' in the attribute dictionary of the LogRecord. If these are missing, the message will not be logged because a string formatting exception will occur. So in this case, you always need to pass the *extra* dictionary with these keys. While this might be annoying, this feature is intended for use in specialized circumstances, such as multi-threaded servers where the same code executes in many contexts, and interesting conditions which arise are dependent on this context (such as remote client IP address and authenticated user name, in the above example). In such circumstances, it is likely that specialized :class:`Formatter`\ s would be used with particular :class:`Handler`\ s. .. versionadded:: 3.2 The *stack_info* parameter was added. .. function:: info(msg, *args, **kwargs) Logs a message with level :const:`INFO` on the root logger. The arguments are interpreted as for :func:`debug`. .. function:: warning(msg, *args, **kwargs) Logs a message with level :const:`WARNING` on the root logger. The arguments are interpreted as for :func:`debug`. .. note:: There is an obsolete function ``warn`` which is functionally identical to ``warning``. As ``warn`` is deprecated, please do not use it - use ``warning`` instead. .. function:: error(msg, *args, **kwargs) Logs a message with level :const:`ERROR` on the root logger. The arguments are interpreted as for :func:`debug`. .. function:: critical(msg, *args, **kwargs) Logs a message with level :const:`CRITICAL` on the root logger. The arguments are interpreted as for :func:`debug`. .. function:: exception(msg, *args, **kwargs) Logs a message with level :const:`ERROR` on the root logger. The arguments are interpreted as for :func:`debug`. Exception info is added to the logging message. This function should only be called from an exception handler. .. function:: log(level, msg, *args, **kwargs) Logs a message with level *level* on the root logger. The other arguments are interpreted as for :func:`debug`. .. note:: The above module-level convenience functions, which delegate to the root logger, call :func:`basicConfig` to ensure that at least one handler is available. Because of this, they should *not* be used in threads, in versions of Python earlier than 2.7.1 and 3.2, unless at least one handler has been added to the root logger *before* the threads are started. In earlier versions of Python, due to a thread safety shortcoming in :func:`basicConfig`, this can (under rare circumstances) lead to handlers being added multiple times to the root logger, which can in turn lead to multiple messages for the same event. .. function:: disable(lvl=CRITICAL) Provides an overriding level *lvl* for all loggers which takes precedence over the logger's own level. When the need arises to temporarily throttle logging output down across the whole application, this function can be useful. Its effect is to disable all logging calls of severity *lvl* and below, so that if you call it with a value of INFO, then all INFO and DEBUG events would be discarded, whereas those of severity WARNING and above would be processed according to the logger's effective level. If ``logging.disable(logging.NOTSET)`` is called, it effectively removes this overriding level, so that logging output again depends on the effective levels of individual loggers. Note that if you have defined any custom logging level higher than ``CRITICAL`` (this is not recommended), you won't be able to rely on the default value for the *lvl* parameter, but will have to explicitly supply a suitable value. .. versionchanged:: 3.7 The *lvl* parameter was defaulted to level ``CRITICAL``. See Issue #28524 for more information about this change. .. function:: addLevelName(lvl, levelName) Associates level *lvl* with text *levelName* in an internal dictionary, which is used to map numeric levels to a textual representation, for example when a :class:`Formatter` formats a message. This function can also be used to define your own levels. The only constraints are that all levels used must be registered using this function, levels should be positive integers and they should increase in increasing order of severity. .. note:: If you are thinking of defining your own levels, please see the section on :ref:`custom-levels`. .. function:: getLevelName(lvl) Returns the textual representation of logging level *lvl*. If the level is one of the predefined levels :const:`CRITICAL`, :const:`ERROR`, :const:`WARNING`, :const:`INFO` or :const:`DEBUG` then you get the corresponding string. If you have associated levels with names using :func:`addLevelName` then the name you have associated with *lvl* is returned. If a numeric value corresponding to one of the defined levels is passed in, the corresponding string representation is returned. Otherwise, the string 'Level %s' % lvl is returned. .. note:: Levels are internally integers (as they need to be compared in the logging logic). This function is used to convert between an integer level and the level name displayed in the formatted log output by means of the ``%(levelname)s`` format specifier (see :ref:`logrecord-attributes`). .. versionchanged:: 3.4 In Python versions earlier than 3.4, this function could also be passed a text level, and would return the corresponding numeric value of the level. This undocumented behaviour was considered a mistake, and was removed in Python 3.4, but reinstated in 3.4.2 due to retain backward compatibility. .. function:: makeLogRecord(attrdict) Creates and returns a new :class:`LogRecord` instance whose attributes are defined by *attrdict*. This function is useful for taking a pickled :class:`LogRecord` attribute dictionary, sent over a socket, and reconstituting it as a :class:`LogRecord` instance at the receiving end. .. function:: basicConfig(**kwargs) Does basic configuration for the logging system by creating a :class:`StreamHandler` with a default :class:`Formatter` and adding it to the root logger. The functions :func:`debug`, :func:`info`, :func:`warning`, :func:`error` and :func:`critical` will call :func:`basicConfig` automatically if no handlers are defined for the root logger. This function does nothing if the root logger already has handlers configured for it. .. note:: This function should be called from the main thread before other threads are started. In versions of Python prior to 2.7.1 and 3.2, if this function is called from multiple threads, it is possible (in rare circumstances) that a handler will be added to the root logger more than once, leading to unexpected results such as messages being duplicated in the log. The following keyword arguments are supported. .. tabularcolumns:: |l|L| +--------------+---------------------------------------------+ | Format | Description | +==============+=============================================+ | *filename* | Specifies that a FileHandler be created, | | | using the specified filename, rather than a | | | StreamHandler. | +--------------+---------------------------------------------+ | *filemode* | If *filename* is specified, open the file | | | in this :ref:`mode `. Defaults | | | to ``'a'``. | +--------------+---------------------------------------------+ | *format* | Use the specified format string for the | | | handler. | +--------------+---------------------------------------------+ | *datefmt* | Use the specified date/time format, as | | | accepted by :func:`time.strftime`. | +--------------+---------------------------------------------+ | *style* | If *format* is specified, use this style | | | for the format string. One of ``'%'``, | | | ``'{'`` or ``'$'`` for :ref:`printf-style | | | `, | | | :meth:`str.format` or | | | :class:`string.Template` respectively. | | | Defaults to ``'%'``. | +--------------+---------------------------------------------+ | *level* | Set the root logger level to the specified | | | :ref:`level `. | +--------------+---------------------------------------------+ | *stream* | Use the specified stream to initialize the | | | StreamHandler. Note that this argument is | | | incompatible with *filename* - if both | | | are present, a ``ValueError`` is raised. | +--------------+---------------------------------------------+ | *handlers* | If specified, this should be an iterable of | | | already created handlers to add to the root | | | logger. Any handlers which don't already | | | have a formatter set will be assigned the | | | default formatter created in this function. | | | Note that this argument is incompatible | | | with *filename* or *stream* - if both | | | are present, a ``ValueError`` is raised. | +--------------+---------------------------------------------+ .. versionchanged:: 3.2 The *style* argument was added. .. versionchanged:: 3.3 The *handlers* argument was added. Additional checks were added to catch situations where incompatible arguments are specified (e.g. *handlers* together with *stream* or *filename*, or *stream* together with *filename*). .. function:: shutdown() Informs the logging system to perform an orderly shutdown by flushing and closing all handlers. This should be called at application exit and no further use of the logging system should be made after this call. .. function:: setLoggerClass(klass) Tells the logging system to use the class *klass* when instantiating a logger. The class should define :meth:`__init__` such that only a name argument is required, and the :meth:`__init__` should call :meth:`Logger.__init__`. This function is typically called before any loggers are instantiated by applications which need to use custom logger behavior. .. function:: setLogRecordFactory(factory) Set a callable which is used to create a :class:`LogRecord`. :param factory: The factory callable to be used to instantiate a log record. .. versionadded:: 3.2 This function has been provided, along with :func:`getLogRecordFactory`, to allow developers more control over how the :class:`LogRecord` representing a logging event is constructed. The factory has the following signature: ``factory(name, level, fn, lno, msg, args, exc_info, func=None, sinfo=None, **kwargs)`` :name: The logger name. :level: The logging level (numeric). :fn: The full pathname of the file where the logging call was made. :lno: The line number in the file where the logging call was made. :msg: The logging message. :args: The arguments for the logging message. :exc_info: An exception tuple, or ``None``. :func: The name of the function or method which invoked the logging call. :sinfo: A stack traceback such as is provided by :func:`traceback.print_stack`, showing the call hierarchy. :kwargs: Additional keyword arguments. Module-Level Attributes ----------------------- .. attribute:: lastResort A "handler of last resort" is available through this attribute. This is a :class:`StreamHandler` writing to ``sys.stderr`` with a level of ``WARNING``, and is used to handle logging events in the absence of any logging configuration. The end result is to just print the message to ``sys.stderr``. This replaces the earlier error message saying that "no handlers could be found for logger XYZ". If you need the earlier behaviour for some reason, ``lastResort`` can be set to ``None``. .. versionadded:: 3.2 Integration with the warnings module ------------------------------------ The :func:`captureWarnings` function can be used to integrate :mod:`logging` with the :mod:`warnings` module. .. function:: captureWarnings(capture) This function is used to turn the capture of warnings by logging on and off. If *capture* is ``True``, warnings issued by the :mod:`warnings` module will be redirected to the logging system. Specifically, a warning will be formatted using :func:`warnings.formatwarning` and the resulting string logged to a logger named ``'py.warnings'`` with a severity of :const:`WARNING`. If *capture* is ``False``, the redirection of warnings to the logging system will stop, and warnings will be redirected to their original destinations (i.e. those in effect before ``captureWarnings(True)`` was called). .. seealso:: Module :mod:`logging.config` Configuration API for the logging module. Module :mod:`logging.handlers` Useful handlers included with the logging module. :pep:`282` - A Logging System The proposal which described this feature for inclusion in the Python standard library. `Original Python logging package `_ This is the original source for the :mod:`logging` package. The version of the package available from this site is suitable for use with Python 1.5.2, 2.1.x and 2.2.x, which do not include the :mod:`logging` package in the standard library. PK 3]Xܴގlibrary/xdrlib.rst.txtnu[:mod:`xdrlib` --- Encode and decode XDR data ============================================ .. module:: xdrlib :synopsis: Encoders and decoders for the External Data Representation (XDR). **Source code:** :source:`Lib/xdrlib.py` .. index:: single: XDR single: External Data Representation -------------- The :mod:`xdrlib` module supports the External Data Representation Standard as described in :rfc:`1014`, written by Sun Microsystems, Inc. June 1987. It supports most of the data types described in the RFC. The :mod:`xdrlib` module defines two classes, one for packing variables into XDR representation, and another for unpacking from XDR representation. There are also two exception classes. .. class:: Packer() :class:`Packer` is the class for packing data into XDR representation. The :class:`Packer` class is instantiated with no arguments. .. class:: Unpacker(data) ``Unpacker`` is the complementary class which unpacks XDR data values from a string buffer. The input buffer is given as *data*. .. seealso:: :rfc:`1014` - XDR: External Data Representation Standard This RFC defined the encoding of data which was XDR at the time this module was originally written. It has apparently been obsoleted by :rfc:`1832`. :rfc:`1832` - XDR: External Data Representation Standard Newer RFC that provides a revised definition of XDR. .. _xdr-packer-objects: Packer Objects -------------- :class:`Packer` instances have the following methods: .. method:: Packer.get_buffer() Returns the current pack buffer as a string. .. method:: Packer.reset() Resets the pack buffer to the empty string. In general, you can pack any of the most common XDR data types by calling the appropriate ``pack_type()`` method. Each method takes a single argument, the value to pack. The following simple data type packing methods are supported: :meth:`pack_uint`, :meth:`pack_int`, :meth:`pack_enum`, :meth:`pack_bool`, :meth:`pack_uhyper`, and :meth:`pack_hyper`. .. method:: Packer.pack_float(value) Packs the single-precision floating point number *value*. .. method:: Packer.pack_double(value) Packs the double-precision floating point number *value*. The following methods support packing strings, bytes, and opaque data: .. method:: Packer.pack_fstring(n, s) Packs a fixed length string, *s*. *n* is the length of the string but it is *not* packed into the data buffer. The string is padded with null bytes if necessary to guaranteed 4 byte alignment. .. method:: Packer.pack_fopaque(n, data) Packs a fixed length opaque data stream, similarly to :meth:`pack_fstring`. .. method:: Packer.pack_string(s) Packs a variable length string, *s*. The length of the string is first packed as an unsigned integer, then the string data is packed with :meth:`pack_fstring`. .. method:: Packer.pack_opaque(data) Packs a variable length opaque data string, similarly to :meth:`pack_string`. .. method:: Packer.pack_bytes(bytes) Packs a variable length byte stream, similarly to :meth:`pack_string`. The following methods support packing arrays and lists: .. method:: Packer.pack_list(list, pack_item) Packs a *list* of homogeneous items. This method is useful for lists with an indeterminate size; i.e. the size is not available until the entire list has been walked. For each item in the list, an unsigned integer ``1`` is packed first, followed by the data value from the list. *pack_item* is the function that is called to pack the individual item. At the end of the list, an unsigned integer ``0`` is packed. For example, to pack a list of integers, the code might appear like this:: import xdrlib p = xdrlib.Packer() p.pack_list([1, 2, 3], p.pack_int) .. method:: Packer.pack_farray(n, array, pack_item) Packs a fixed length list (*array*) of homogeneous items. *n* is the length of the list; it is *not* packed into the buffer, but a :exc:`ValueError` exception is raised if ``len(array)`` is not equal to *n*. As above, *pack_item* is the function used to pack each element. .. method:: Packer.pack_array(list, pack_item) Packs a variable length *list* of homogeneous items. First, the length of the list is packed as an unsigned integer, then each element is packed as in :meth:`pack_farray` above. .. _xdr-unpacker-objects: Unpacker Objects ---------------- The :class:`Unpacker` class offers the following methods: .. method:: Unpacker.reset(data) Resets the string buffer with the given *data*. .. method:: Unpacker.get_position() Returns the current unpack position in the data buffer. .. method:: Unpacker.set_position(position) Sets the data buffer unpack position to *position*. You should be careful about using :meth:`get_position` and :meth:`set_position`. .. method:: Unpacker.get_buffer() Returns the current unpack data buffer as a string. .. method:: Unpacker.done() Indicates unpack completion. Raises an :exc:`Error` exception if all of the data has not been unpacked. In addition, every data type that can be packed with a :class:`Packer`, can be unpacked with an :class:`Unpacker`. Unpacking methods are of the form ``unpack_type()``, and take no arguments. They return the unpacked object. .. method:: Unpacker.unpack_float() Unpacks a single-precision floating point number. .. method:: Unpacker.unpack_double() Unpacks a double-precision floating point number, similarly to :meth:`unpack_float`. In addition, the following methods unpack strings, bytes, and opaque data: .. method:: Unpacker.unpack_fstring(n) Unpacks and returns a fixed length string. *n* is the number of characters expected. Padding with null bytes to guaranteed 4 byte alignment is assumed. .. method:: Unpacker.unpack_fopaque(n) Unpacks and returns a fixed length opaque data stream, similarly to :meth:`unpack_fstring`. .. method:: Unpacker.unpack_string() Unpacks and returns a variable length string. The length of the string is first unpacked as an unsigned integer, then the string data is unpacked with :meth:`unpack_fstring`. .. method:: Unpacker.unpack_opaque() Unpacks and returns a variable length opaque data string, similarly to :meth:`unpack_string`. .. method:: Unpacker.unpack_bytes() Unpacks and returns a variable length byte stream, similarly to :meth:`unpack_string`. The following methods support unpacking arrays and lists: .. method:: Unpacker.unpack_list(unpack_item) Unpacks and returns a list of homogeneous items. The list is unpacked one element at a time by first unpacking an unsigned integer flag. If the flag is ``1``, then the item is unpacked and appended to the list. A flag of ``0`` indicates the end of the list. *unpack_item* is the function that is called to unpack the items. .. method:: Unpacker.unpack_farray(n, unpack_item) Unpacks and returns (as a list) a fixed length array of homogeneous items. *n* is number of list elements to expect in the buffer. As above, *unpack_item* is the function used to unpack each element. .. method:: Unpacker.unpack_array(unpack_item) Unpacks and returns a variable length *list* of homogeneous items. First, the length of the list is unpacked as an unsigned integer, then each element is unpacked as in :meth:`unpack_farray` above. .. _xdr-exceptions: Exceptions ---------- Exceptions in this module are coded as class instances: .. exception:: Error The base exception class. :exc:`Error` has a single public attribute :attr:`msg` containing the description of the error. .. exception:: ConversionError Class derived from :exc:`Error`. Contains no additional instance variables. Here is an example of how you would catch one of these exceptions:: import xdrlib p = xdrlib.Packer() try: p.pack_double(8.01) except xdrlib.ConversionError as instance: print('packing the double failed:', instance.msg) PK 3]aE 7 7library/email.parser.rst.txtnu[:mod:`email.parser`: Parsing email messages ------------------------------------------- .. module:: email.parser :synopsis: Parse flat text email messages to produce a message object structure. **Source code:** :source:`Lib/email/parser.py` -------------- Message object structures can be created in one of two ways: they can be created from whole cloth by creating an :class:`~email.message.EmailMessage` object, adding headers using the dictionary interface, and adding payload(s) using :meth:`~email.message.EmailMessage.set_content` and related methods, or they can be created by parsing a serialized representation of the email message. The :mod:`email` package provides a standard parser that understands most email document structures, including MIME documents. You can pass the parser a bytes, string or file object, and the parser will return to you the root :class:`~email.message.EmailMessage` instance of the object structure. For simple, non-MIME messages the payload of this root object will likely be a string containing the text of the message. For MIME messages, the root object will return ``True`` from its :meth:`~email.message.EmailMessage.is_multipart` method, and the subparts can be accessed via the payload manipulation methods, such as :meth:`~email.message.EmailMessage.get_body`, :meth:`~email.message.EmailMessage.iter_parts`, and :meth:`~email.message.EmailMessage.walk`. There are actually two parser interfaces available for use, the :class:`Parser` API and the incremental :class:`FeedParser` API. The :class:`Parser` API is most useful if you have the entire text of the message in memory, or if the entire message lives in a file on the file system. :class:`FeedParser` is more appropriate when you are reading the message from a stream which might block waiting for more input (such as reading an email message from a socket). The :class:`FeedParser` can consume and parse the message incrementally, and only returns the root object when you close the parser. Note that the parser can be extended in limited ways, and of course you can implement your own parser completely from scratch. All of the logic that connects the :mod:`email` package's bundled parser and the :class:`~email.message.EmailMessage` class is embodied in the :mod:`policy` class, so a custom parser can create message object trees any way it finds necessary by implementing custom versions of the appropriate :mod:`policy` methods. FeedParser API ^^^^^^^^^^^^^^ The :class:`BytesFeedParser`, imported from the :mod:`email.feedparser` module, provides an API that is conducive to incremental parsing of email messages, such as would be necessary when reading the text of an email message from a source that can block (such as a socket). The :class:`BytesFeedParser` can of course be used to parse an email message fully contained in a :term:`bytes-like object`, string, or file, but the :class:`BytesParser` API may be more convenient for such use cases. The semantics and results of the two parser APIs are identical. The :class:`BytesFeedParser`'s API is simple; you create an instance, feed it a bunch of bytes until there's no more to feed it, then close the parser to retrieve the root message object. The :class:`BytesFeedParser` is extremely accurate when parsing standards-compliant messages, and it does a very good job of parsing non-compliant messages, providing information about how a message was deemed broken. It will populate a message object's :attr:`~email.message.EmailMessage.defects` attribute with a list of any problems it found in a message. See the :mod:`email.errors` module for the list of defects that it can find. Here is the API for the :class:`BytesFeedParser`: .. class:: BytesFeedParser(_factory=None, *, policy=policy.compat32) Create a :class:`BytesFeedParser` instance. Optional *_factory* is a no-argument callable; if not specified use the :attr:`~email.policy.Policy.message_factory` from the *policy*. Call *_factory* whenever a new message object is needed. If *policy* is specified use the rules it specifies to update the representation of the message. If *policy* is not set, use the :class:`compat32 ` policy, which maintains backward compatibility with the Python 3.2 version of the email package and provides :class:`~email.message.Message` as the default factory. All other policies provide :class:`~email.message.EmailMessage` as the default *_factory*. For more information on what else *policy* controls, see the :mod:`~email.policy` documentation. Note: **The policy keyword should always be specified**; The default will change to :data:`email.policy.default` in a future version of Python. .. versionadded:: 3.2 .. versionchanged:: 3.3 Added the *policy* keyword. .. versionchanged:: 3.6 *_factory* defaults to the policy ``message_factory``. .. method:: feed(data) Feed the parser some more data. *data* should be a :term:`bytes-like object` containing one or more lines. The lines can be partial and the parser will stitch such partial lines together properly. The lines can have any of the three common line endings: carriage return, newline, or carriage return and newline (they can even be mixed). .. method:: close() Complete the parsing of all previously fed data and return the root message object. It is undefined what happens if :meth:`~feed` is called after this method has been called. .. class:: FeedParser(_factory=None, *, policy=policy.compat32) Works like :class:`BytesFeedParser` except that the input to the :meth:`~BytesFeedParser.feed` method must be a string. This is of limited utility, since the only way for such a message to be valid is for it to contain only ASCII text or, if :attr:`~email.policy.Policy.utf8` is ``True``, no binary attachments. .. versionchanged:: 3.3 Added the *policy* keyword. Parser API ^^^^^^^^^^ The :class:`BytesParser` class, imported from the :mod:`email.parser` module, provides an API that can be used to parse a message when the complete contents of the message are available in a :term:`bytes-like object` or file. The :mod:`email.parser` module also provides :class:`Parser` for parsing strings, and header-only parsers, :class:`BytesHeaderParser` and :class:`HeaderParser`, which can be used if you're only interested in the headers of the message. :class:`BytesHeaderParser` and :class:`HeaderParser` can be much faster in these situations, since they do not attempt to parse the message body, instead setting the payload to the raw body. .. class:: BytesParser(_class=None, *, policy=policy.compat32) Create a :class:`BytesParser` instance. The *_class* and *policy* arguments have the same meaning and semantics as the *_factory* and *policy* arguments of :class:`BytesFeedParser`. Note: **The policy keyword should always be specified**; The default will change to :data:`email.policy.default` in a future version of Python. .. versionchanged:: 3.3 Removed the *strict* argument that was deprecated in 2.4. Added the *policy* keyword. .. versionchanged:: 3.6 *_class* defaults to the policy ``message_factory``. .. method:: parse(fp, headersonly=False) Read all the data from the binary file-like object *fp*, parse the resulting bytes, and return the message object. *fp* must support both the :meth:`~io.IOBase.readline` and the :meth:`~io.IOBase.read` methods. The bytes contained in *fp* must be formatted as a block of :rfc:`5322` (or, if :attr:`~email.policy.Policy.utf8` is ``True``, :rfc:`6532`) style headers and header continuation lines, optionally preceded by an envelope header. The header block is terminated either by the end of the data or by a blank line. Following the header block is the body of the message (which may contain MIME-encoded subparts, including subparts with a :mailheader:`Content-Transfer-Encoding` of ``8bit``. Optional *headersonly* is a flag specifying whether to stop parsing after reading the headers or not. The default is ``False``, meaning it parses the entire contents of the file. .. method:: parsebytes(bytes, headersonly=False) Similar to the :meth:`parse` method, except it takes a :term:`bytes-like object` instead of a file-like object. Calling this method on a :term:`bytes-like object` is equivalent to wrapping *bytes* in a :class:`~io.BytesIO` instance first and calling :meth:`parse`. Optional *headersonly* is as with the :meth:`parse` method. .. versionadded:: 3.2 .. class:: BytesHeaderParser(_class=None, *, policy=policy.compat32) Exactly like :class:`BytesParser`, except that *headersonly* defaults to ``True``. .. versionadded:: 3.3 .. class:: Parser(_class=None, *, policy=policy.compat32) This class is parallel to :class:`BytesParser`, but handles string input. .. versionchanged:: 3.3 Removed the *strict* argument. Added the *policy* keyword. .. versionchanged:: 3.6 *_class* defaults to the policy ``message_factory``. .. method:: parse(fp, headersonly=False) Read all the data from the text-mode file-like object *fp*, parse the resulting text, and return the root message object. *fp* must support both the :meth:`~io.TextIOBase.readline` and the :meth:`~io.TextIOBase.read` methods on file-like objects. Other than the text mode requirement, this method operates like :meth:`BytesParser.parse`. .. method:: parsestr(text, headersonly=False) Similar to the :meth:`parse` method, except it takes a string object instead of a file-like object. Calling this method on a string is equivalent to wrapping *text* in a :class:`~io.StringIO` instance first and calling :meth:`parse`. Optional *headersonly* is as with the :meth:`parse` method. .. class:: HeaderParser(_class=None, *, policy=policy.compat32) Exactly like :class:`Parser`, except that *headersonly* defaults to ``True``. Since creating a message object structure from a string or a file object is such a common task, four functions are provided as a convenience. They are available in the top-level :mod:`email` package namespace. .. currentmodule:: email .. function:: message_from_bytes(s, _class=None, *, policy=policy.compat32) Return a message object structure from a :term:`bytes-like object`. This is equivalent to ``BytesParser().parsebytes(s)``. Optional *_class* and *strict* are interpreted as with the :class:`~email.parser.BytesParser` class constructor. .. versionadded:: 3.2 .. versionchanged:: 3.3 Removed the *strict* argument. Added the *policy* keyword. .. function:: message_from_binary_file(fp, _class=None, *, \ policy=policy.compat32) Return a message object structure tree from an open binary :term:`file object`. This is equivalent to ``BytesParser().parse(fp)``. *_class* and *policy* are interpreted as with the :class:`~email.parser.BytesParser` class constructor. .. versionadded:: 3.2 .. versionchanged:: 3.3 Removed the *strict* argument. Added the *policy* keyword. .. function:: message_from_string(s, _class=None, *, policy=policy.compat32) Return a message object structure from a string. This is equivalent to ``Parser().parsestr(s)``. *_class* and *policy* are interpreted as with the :class:`~email.parser.Parser` class constructor. .. versionchanged:: 3.3 Removed the *strict* argument. Added the *policy* keyword. .. function:: message_from_file(fp, _class=None, *, policy=policy.compat32) Return a message object structure tree from an open :term:`file object`. This is equivalent to ``Parser().parse(fp)``. *_class* and *policy* are interpreted as with the :class:`~email.parser.Parser` class constructor. .. versionchanged:: 3.3 Removed the *strict* argument. Added the *policy* keyword. .. versionchanged:: 3.6 *_class* defaults to the policy ``message_factory``. Here's an example of how you might use :func:`message_from_bytes` at an interactive Python prompt:: >>> import email >>> msg = email.message_from_bytes(myBytes) # doctest: +SKIP Additional notes ^^^^^^^^^^^^^^^^ Here are some notes on the parsing semantics: * Most non-\ :mimetype:`multipart` type messages are parsed as a single message object with a string payload. These objects will return ``False`` for :meth:`~email.message.EmailMessage.is_multipart`, and :meth:`~email.message.EmailMessage.iter_parts` will yield an empty list. * All :mimetype:`multipart` type messages will be parsed as a container message object with a list of sub-message objects for their payload. The outer container message will return ``True`` for :meth:`~email.message.EmailMessage.is_multipart`, and :meth:`~email.message.EmailMessage.iter_parts` will yield a list of subparts. * Most messages with a content type of :mimetype:`message/\*` (such as :mimetype:`message/delivery-status` and :mimetype:`message/rfc822`) will also be parsed as container object containing a list payload of length 1. Their :meth:`~email.message.EmailMessage.is_multipart` method will return ``True``. The single element yielded by :meth:`~email.message.EmailMessage.iter_parts` will be a sub-message object. * Some non-standards-compliant messages may not be internally consistent about their :mimetype:`multipart`\ -edness. Such messages may have a :mailheader:`Content-Type` header of type :mimetype:`multipart`, but their :meth:`~email.message.EmailMessage.is_multipart` method may return ``False``. If such messages were parsed with the :class:`~email.parser.FeedParser`, they will have an instance of the :class:`~email.errors.MultipartInvariantViolationDefect` class in their *defects* attribute list. See :mod:`email.errors` for details. PK 3]:fЎlibrary/binary.rst.txtnu[.. _binaryservices: ******************** Binary Data Services ******************** The modules described in this chapter provide some basic services operations for manipulation of binary data. Other operations on binary data, specifically in relation to file formats and network protocols, are described in the relevant sections. Some libraries described under :ref:`textservices` also work with either ASCII-compatible binary formats (for example, :mod:`re`) or all binary data (for example, :mod:`difflib`). In addition, see the documentation for Python's built-in binary data types in :ref:`binaryseq`. .. toctree:: struct.rst codecs.rst PK 3]߭e"library/asyncio-eventloops.rst.txtnu[.. currentmodule:: asyncio Event loops =========== **Source code:** :source:`Lib/asyncio/events.py` Event loop functions -------------------- The following functions are convenient shortcuts to accessing the methods of the global policy. Note that this provides access to the default policy, unless an alternative policy was set by calling :func:`set_event_loop_policy` earlier in the execution of the process. .. function:: get_event_loop() Equivalent to calling ``get_event_loop_policy().get_event_loop()``. .. function:: set_event_loop(loop) Equivalent to calling ``get_event_loop_policy().set_event_loop(loop)``. .. function:: new_event_loop() Equivalent to calling ``get_event_loop_policy().new_event_loop()``. .. _asyncio-event-loops: Available event loops --------------------- asyncio currently provides two implementations of event loops: :class:`SelectorEventLoop` and :class:`ProactorEventLoop`. .. class:: SelectorEventLoop Event loop based on the :mod:`selectors` module. Subclass of :class:`AbstractEventLoop`. Use the most efficient selector available on the platform. On Windows, only sockets are supported (ex: pipes are not supported): see the `MSDN documentation of select `_. .. class:: ProactorEventLoop Proactor event loop for Windows using "I/O Completion Ports" aka IOCP. Subclass of :class:`AbstractEventLoop`. Availability: Windows. .. seealso:: `MSDN documentation on I/O Completion Ports `_. Example to use a :class:`ProactorEventLoop` on Windows:: import asyncio, sys if sys.platform == 'win32': loop = asyncio.ProactorEventLoop() asyncio.set_event_loop(loop) .. _asyncio-platform-support: Platform support ---------------- The :mod:`asyncio` module has been designed to be portable, but each platform still has subtle differences and may not support all :mod:`asyncio` features. Windows ^^^^^^^ Common limits of Windows event loops: - :meth:`~AbstractEventLoop.create_unix_connection` and :meth:`~AbstractEventLoop.create_unix_server` are not supported: the socket family :data:`socket.AF_UNIX` is specific to UNIX - :meth:`~AbstractEventLoop.add_signal_handler` and :meth:`~AbstractEventLoop.remove_signal_handler` are not supported - :meth:`EventLoopPolicy.set_child_watcher` is not supported. :class:`ProactorEventLoop` supports subprocesses. It has only one implementation to watch child processes, there is no need to configure it. :class:`SelectorEventLoop` specific limits: - :class:`~selectors.SelectSelector` is used which only supports sockets and is limited to 512 sockets. - :meth:`~AbstractEventLoop.add_reader` and :meth:`~AbstractEventLoop.add_writer` only accept file descriptors of sockets - Pipes are not supported (ex: :meth:`~AbstractEventLoop.connect_read_pipe`, :meth:`~AbstractEventLoop.connect_write_pipe`) - :ref:`Subprocesses ` are not supported (ex: :meth:`~AbstractEventLoop.subprocess_exec`, :meth:`~AbstractEventLoop.subprocess_shell`) :class:`ProactorEventLoop` specific limits: - :meth:`~AbstractEventLoop.create_datagram_endpoint` (UDP) is not supported - :meth:`~AbstractEventLoop.add_reader` and :meth:`~AbstractEventLoop.add_writer` are not supported The resolution of the monotonic clock on Windows is usually around 15.6 msec. The best resolution is 0.5 msec. The resolution depends on the hardware (availability of `HPET `_) and on the Windows configuration. See :ref:`asyncio delayed calls `. .. versionchanged:: 3.5 :class:`ProactorEventLoop` now supports SSL. Mac OS X ^^^^^^^^ Character devices like PTY are only well supported since Mavericks (Mac OS 10.9). They are not supported at all on Mac OS 10.5 and older. On Mac OS 10.6, 10.7 and 10.8, the default event loop is :class:`SelectorEventLoop` which uses :class:`selectors.KqueueSelector`. :class:`selectors.KqueueSelector` does not support character devices on these versions. The :class:`SelectorEventLoop` can be used with :class:`~selectors.SelectSelector` or :class:`~selectors.PollSelector` to support character devices on these versions of Mac OS X. Example:: import asyncio import selectors selector = selectors.SelectSelector() loop = asyncio.SelectorEventLoop(selector) asyncio.set_event_loop(loop) Event loop policies and the default policy ------------------------------------------ Event loop management is abstracted with a *policy* pattern, to provide maximal flexibility for custom platforms and frameworks. Throughout the execution of a process, a single global policy object manages the event loops available to the process based on the calling context. A policy is an object implementing the :class:`AbstractEventLoopPolicy` interface. For most users of :mod:`asyncio`, policies never have to be dealt with explicitly, since the default global policy is sufficient (see below). The module-level functions :func:`get_event_loop` and :func:`set_event_loop` provide convenient access to event loops managed by the default policy. Event loop policy interface --------------------------- An event loop policy must implement the following interface: .. class:: AbstractEventLoopPolicy Event loop policy. .. method:: get_event_loop() Get the event loop for the current context. Returns an event loop object implementing the :class:`AbstractEventLoop` interface. In case called from coroutine, it returns the currently running event loop. Raises an exception in case no event loop has been set for the current context and the current policy does not specify to create one. It must never return ``None``. .. versionchanged:: 3.6 .. method:: set_event_loop(loop) Set the event loop for the current context to *loop*. .. method:: new_event_loop() Create and return a new event loop object according to this policy's rules. If there's need to set this loop as the event loop for the current context, :meth:`set_event_loop` must be called explicitly. The default policy defines context as the current thread, and manages an event loop per thread that interacts with :mod:`asyncio`. If the current thread doesn't already have an event loop associated with it, the default policy's :meth:`~AbstractEventLoopPolicy.get_event_loop` method creates one when called from the main thread, but raises :exc:`RuntimeError` otherwise. Access to the global loop policy -------------------------------- .. function:: get_event_loop_policy() Get the current event loop policy. .. function:: set_event_loop_policy(policy) Set the current event loop policy. If *policy* is ``None``, the default policy is restored. Customizing the event loop policy --------------------------------- To implement a new event loop policy, it is recommended you subclass the concrete default event loop policy :class:`DefaultEventLoopPolicy` and override the methods for which you want to change behavior, for example:: class MyEventLoopPolicy(asyncio.DefaultEventLoopPolicy): def get_event_loop(self): """Get the event loop. This may be None or an instance of EventLoop. """ loop = super().get_event_loop() # Do something with loop ... return loop asyncio.set_event_loop_policy(MyEventLoopPolicy()) PK 3]3 3 library/pty.rst.txtnu[:mod:`pty` --- Pseudo-terminal utilities ======================================== .. module:: pty :platform: Linux :synopsis: Pseudo-Terminal Handling for Linux. .. moduleauthor:: Steen Lumholt .. sectionauthor:: Moshe Zadka **Source code:** :source:`Lib/pty.py` -------------- The :mod:`pty` module defines operations for handling the pseudo-terminal concept: starting another process and being able to write to and read from its controlling terminal programmatically. Because pseudo-terminal handling is highly platform dependent, there is code to do it only for Linux. (The Linux code is supposed to work on other platforms, but hasn't been tested yet.) The :mod:`pty` module defines the following functions: .. function:: fork() Fork. Connect the child's controlling terminal to a pseudo-terminal. Return value is ``(pid, fd)``. Note that the child gets *pid* 0, and the *fd* is *invalid*. The parent's return value is the *pid* of the child, and *fd* is a file descriptor connected to the child's controlling terminal (and also to the child's standard input and output). .. function:: openpty() Open a new pseudo-terminal pair, using :func:`os.openpty` if possible, or emulation code for generic Unix systems. Return a pair of file descriptors ``(master, slave)``, for the master and the slave end, respectively. .. function:: spawn(argv[, master_read[, stdin_read]]) Spawn a process, and connect its controlling terminal with the current process's standard io. This is often used to baffle programs which insist on reading from the controlling terminal. The functions *master_read* and *stdin_read* should be functions which read from a file descriptor. The defaults try to read 1024 bytes each time they are called. .. versionchanged:: 3.4 :func:`spawn` now returns the status value from :func:`os.waitpid` on the child process. Example ------- .. sectionauthor:: Steen Lumholt The following program acts like the Unix command :manpage:`script(1)`, using a pseudo-terminal to record all input and output of a terminal session in a "typescript". :: import argparse import os import pty import sys import time parser = argparse.ArgumentParser() parser.add_argument('-a', dest='append', action='store_true') parser.add_argument('-p', dest='use_python', action='store_true') parser.add_argument('filename', nargs='?', default='typescript') options = parser.parse_args() shell = sys.executable if options.use_python else os.environ.get('SHELL', 'sh') filename = options.filename mode = 'ab' if options.append else 'wb' with open(filename, mode) as script: def read(fd): data = os.read(fd, 1024) script.write(data) return data print('Script started, file is', filename) script.write(('Script started on %s\n' % time.asctime()).encode()) pty.spawn(shell, read) script.write(('Script done on %s\n' % time.asctime()).encode()) print('Script done, file is', filename) PK 3]ehlibrary/fileformats.rst.txtnu[.. _fileformats: ************ File Formats ************ The modules described in this chapter parse various miscellaneous file formats that aren't markup languages and are not related to e-mail. .. toctree:: csv.rst configparser.rst netrc.rst xdrlib.rst plistlib.rst PK 3]҅m,;,;library/parser.rst.txtnu[:mod:`parser` --- Access Python parse trees =========================================== .. module:: parser :synopsis: Access parse trees for Python source code. .. moduleauthor:: Fred L. Drake, Jr. .. sectionauthor:: Fred L. Drake, Jr. .. Copyright 1995 Virginia Polytechnic Institute and State University and Fred L. Drake, Jr. This copyright notice must be distributed on all copies, but this document otherwise may be distributed as part of the Python distribution. No fee may be charged for this document in any representation, either on paper or electronically. This restriction does not affect other elements in a distributed package in any way. .. index:: single: parsing; Python source code -------------- The :mod:`parser` module provides an interface to Python's internal parser and byte-code compiler. The primary purpose for this interface is to allow Python code to edit the parse tree of a Python expression and create executable code from this. This is better than trying to parse and modify an arbitrary Python code fragment as a string because parsing is performed in a manner identical to the code forming the application. It is also faster. .. note:: From Python 2.5 onward, it's much more convenient to cut in at the Abstract Syntax Tree (AST) generation and compilation stage, using the :mod:`ast` module. There are a few things to note about this module which are important to making use of the data structures created. This is not a tutorial on editing the parse trees for Python code, but some examples of using the :mod:`parser` module are presented. Most importantly, a good understanding of the Python grammar processed by the internal parser is required. For full information on the language syntax, refer to :ref:`reference-index`. The parser itself is created from a grammar specification defined in the file :file:`Grammar/Grammar` in the standard Python distribution. The parse trees stored in the ST objects created by this module are the actual output from the internal parser when created by the :func:`expr` or :func:`suite` functions, described below. The ST objects created by :func:`sequence2st` faithfully simulate those structures. Be aware that the values of the sequences which are considered "correct" will vary from one version of Python to another as the formal grammar for the language is revised. However, transporting code from one Python version to another as source text will always allow correct parse trees to be created in the target version, with the only restriction being that migrating to an older version of the interpreter will not support more recent language constructs. The parse trees are not typically compatible from one version to another, whereas source code has always been forward-compatible. Each element of the sequences returned by :func:`st2list` or :func:`st2tuple` has a simple form. Sequences representing non-terminal elements in the grammar always have a length greater than one. The first element is an integer which identifies a production in the grammar. These integers are given symbolic names in the C header file :file:`Include/graminit.h` and the Python module :mod:`symbol`. Each additional element of the sequence represents a component of the production as recognized in the input string: these are always sequences which have the same form as the parent. An important aspect of this structure which should be noted is that keywords used to identify the parent node type, such as the keyword :keyword:`if` in an :const:`if_stmt`, are included in the node tree without any special treatment. For example, the :keyword:`if` keyword is represented by the tuple ``(1, 'if')``, where ``1`` is the numeric value associated with all :const:`NAME` tokens, including variable and function names defined by the user. In an alternate form returned when line number information is requested, the same token might be represented as ``(1, 'if', 12)``, where the ``12`` represents the line number at which the terminal symbol was found. Terminal elements are represented in much the same way, but without any child elements and the addition of the source text which was identified. The example of the :keyword:`if` keyword above is representative. The various types of terminal symbols are defined in the C header file :file:`Include/token.h` and the Python module :mod:`token`. The ST objects are not required to support the functionality of this module, but are provided for three purposes: to allow an application to amortize the cost of processing complex parse trees, to provide a parse tree representation which conserves memory space when compared to the Python list or tuple representation, and to ease the creation of additional modules in C which manipulate parse trees. A simple "wrapper" class may be created in Python to hide the use of ST objects. The :mod:`parser` module defines functions for a few distinct purposes. The most important purposes are to create ST objects and to convert ST objects to other representations such as parse trees and compiled code objects, but there are also functions which serve to query the type of parse tree represented by an ST object. .. seealso:: Module :mod:`symbol` Useful constants representing internal nodes of the parse tree. Module :mod:`token` Useful constants representing leaf nodes of the parse tree and functions for testing node values. .. _creating-sts: Creating ST Objects ------------------- ST objects may be created from source code or from a parse tree. When creating an ST object from source, different functions are used to create the ``'eval'`` and ``'exec'`` forms. .. function:: expr(source) The :func:`expr` function parses the parameter *source* as if it were an input to ``compile(source, 'file.py', 'eval')``. If the parse succeeds, an ST object is created to hold the internal parse tree representation, otherwise an appropriate exception is raised. .. function:: suite(source) The :func:`suite` function parses the parameter *source* as if it were an input to ``compile(source, 'file.py', 'exec')``. If the parse succeeds, an ST object is created to hold the internal parse tree representation, otherwise an appropriate exception is raised. .. function:: sequence2st(sequence) This function accepts a parse tree represented as a sequence and builds an internal representation if possible. If it can validate that the tree conforms to the Python grammar and all nodes are valid node types in the host version of Python, an ST object is created from the internal representation and returned to the called. If there is a problem creating the internal representation, or if the tree cannot be validated, a :exc:`ParserError` exception is raised. An ST object created this way should not be assumed to compile correctly; normal exceptions raised by compilation may still be initiated when the ST object is passed to :func:`compilest`. This may indicate problems not related to syntax (such as a :exc:`MemoryError` exception), but may also be due to constructs such as the result of parsing ``del f(0)``, which escapes the Python parser but is checked by the bytecode compiler. Sequences representing terminal tokens may be represented as either two-element lists of the form ``(1, 'name')`` or as three-element lists of the form ``(1, 'name', 56)``. If the third element is present, it is assumed to be a valid line number. The line number may be specified for any subset of the terminal symbols in the input tree. .. function:: tuple2st(sequence) This is the same function as :func:`sequence2st`. This entry point is maintained for backward compatibility. .. _converting-sts: Converting ST Objects --------------------- ST objects, regardless of the input used to create them, may be converted to parse trees represented as list- or tuple- trees, or may be compiled into executable code objects. Parse trees may be extracted with or without line numbering information. .. function:: st2list(st, line_info=False, col_info=False) This function accepts an ST object from the caller in *st* and returns a Python list representing the equivalent parse tree. The resulting list representation can be used for inspection or the creation of a new parse tree in list form. This function does not fail so long as memory is available to build the list representation. If the parse tree will only be used for inspection, :func:`st2tuple` should be used instead to reduce memory consumption and fragmentation. When the list representation is required, this function is significantly faster than retrieving a tuple representation and converting that to nested lists. If *line_info* is true, line number information will be included for all terminal tokens as a third element of the list representing the token. Note that the line number provided specifies the line on which the token *ends*. This information is omitted if the flag is false or omitted. .. function:: st2tuple(st, line_info=False, col_info=False) This function accepts an ST object from the caller in *st* and returns a Python tuple representing the equivalent parse tree. Other than returning a tuple instead of a list, this function is identical to :func:`st2list`. If *line_info* is true, line number information will be included for all terminal tokens as a third element of the list representing the token. This information is omitted if the flag is false or omitted. .. function:: compilest(st, filename='') .. index:: builtin: exec builtin: eval The Python byte compiler can be invoked on an ST object to produce code objects which can be used as part of a call to the built-in :func:`exec` or :func:`eval` functions. This function provides the interface to the compiler, passing the internal parse tree from *st* to the parser, using the source file name specified by the *filename* parameter. The default value supplied for *filename* indicates that the source was an ST object. Compiling an ST object may result in exceptions related to compilation; an example would be a :exc:`SyntaxError` caused by the parse tree for ``del f(0)``: this statement is considered legal within the formal grammar for Python but is not a legal language construct. The :exc:`SyntaxError` raised for this condition is actually generated by the Python byte-compiler normally, which is why it can be raised at this point by the :mod:`parser` module. Most causes of compilation failure can be diagnosed programmatically by inspection of the parse tree. .. _querying-sts: Queries on ST Objects --------------------- Two functions are provided which allow an application to determine if an ST was created as an expression or a suite. Neither of these functions can be used to determine if an ST was created from source code via :func:`expr` or :func:`suite` or from a parse tree via :func:`sequence2st`. .. function:: isexpr(st) .. index:: builtin: compile When *st* represents an ``'eval'`` form, this function returns true, otherwise it returns false. This is useful, since code objects normally cannot be queried for this information using existing built-in functions. Note that the code objects created by :func:`compilest` cannot be queried like this either, and are identical to those created by the built-in :func:`compile` function. .. function:: issuite(st) This function mirrors :func:`isexpr` in that it reports whether an ST object represents an ``'exec'`` form, commonly known as a "suite." It is not safe to assume that this function is equivalent to ``not isexpr(st)``, as additional syntactic fragments may be supported in the future. .. _st-errors: Exceptions and Error Handling ----------------------------- The parser module defines a single exception, but may also pass other built-in exceptions from other portions of the Python runtime environment. See each function for information about the exceptions it can raise. .. exception:: ParserError Exception raised when a failure occurs within the parser module. This is generally produced for validation failures rather than the built-in :exc:`SyntaxError` raised during normal parsing. The exception argument is either a string describing the reason of the failure or a tuple containing a sequence causing the failure from a parse tree passed to :func:`sequence2st` and an explanatory string. Calls to :func:`sequence2st` need to be able to handle either type of exception, while calls to other functions in the module will only need to be aware of the simple string values. Note that the functions :func:`compilest`, :func:`expr`, and :func:`suite` may raise exceptions which are normally raised by the parsing and compilation process. These include the built in exceptions :exc:`MemoryError`, :exc:`OverflowError`, :exc:`SyntaxError`, and :exc:`SystemError`. In these cases, these exceptions carry all the meaning normally associated with them. Refer to the descriptions of each function for detailed information. .. _st-objects: ST Objects ---------- Ordered and equality comparisons are supported between ST objects. Pickling of ST objects (using the :mod:`pickle` module) is also supported. .. data:: STType The type of the objects returned by :func:`expr`, :func:`suite` and :func:`sequence2st`. ST objects have the following methods: .. method:: ST.compile(filename='') Same as ``compilest(st, filename)``. .. method:: ST.isexpr() Same as ``isexpr(st)``. .. method:: ST.issuite() Same as ``issuite(st)``. .. method:: ST.tolist(line_info=False, col_info=False) Same as ``st2list(st, line_info, col_info)``. .. method:: ST.totuple(line_info=False, col_info=False) Same as ``st2tuple(st, line_info, col_info)``. Example: Emulation of :func:`compile` ------------------------------------- While many useful operations may take place between parsing and bytecode generation, the simplest operation is to do nothing. For this purpose, using the :mod:`parser` module to produce an intermediate data structure is equivalent to the code :: >>> code = compile('a + 5', 'file.py', 'eval') >>> a = 5 >>> eval(code) 10 The equivalent operation using the :mod:`parser` module is somewhat longer, and allows the intermediate internal parse tree to be retained as an ST object:: >>> import parser >>> st = parser.expr('a + 5') >>> code = st.compile('file.py') >>> a = 5 >>> eval(code) 10 An application which needs both ST and code objects can package this code into readily available functions:: import parser def load_suite(source_string): st = parser.suite(source_string) return st, st.compile() def load_expression(source_string): st = parser.expr(source_string) return st, st.compile() PK 3]s9 library/email.iterators.rst.txtnu[:mod:`email.iterators`: Iterators --------------------------------- .. module:: email.iterators :synopsis: Iterate over a message object tree. **Source code:** :source:`Lib/email/iterators.py` -------------- Iterating over a message object tree is fairly easy with the :meth:`Message.walk ` method. The :mod:`email.iterators` module provides some useful higher level iterations over message object trees. .. function:: body_line_iterator(msg, decode=False) This iterates over all the payloads in all the subparts of *msg*, returning the string payloads line-by-line. It skips over all the subpart headers, and it skips over any subpart with a payload that isn't a Python string. This is somewhat equivalent to reading the flat text representation of the message from a file using :meth:`~io.TextIOBase.readline`, skipping over all the intervening headers. Optional *decode* is passed through to :meth:`Message.get_payload `. .. function:: typed_subpart_iterator(msg, maintype='text', subtype=None) This iterates over all the subparts of *msg*, returning only those subparts that match the MIME type specified by *maintype* and *subtype*. Note that *subtype* is optional; if omitted, then subpart MIME type matching is done only with the main type. *maintype* is optional too; it defaults to :mimetype:`text`. Thus, by default :func:`typed_subpart_iterator` returns each subpart that has a MIME type of :mimetype:`text/\*`. The following function has been added as a useful debugging tool. It should *not* be considered part of the supported public interface for the package. .. function:: _structure(msg, fp=None, level=0, include_default=False) Prints an indented representation of the content types of the message object structure. For example: .. testsetup:: import email from email.iterators import _structure somefile = open('../Lib/test/test_email/data/msg_02.txt') .. doctest:: >>> msg = email.message_from_file(somefile) >>> _structure(msg) multipart/mixed text/plain text/plain multipart/digest message/rfc822 text/plain message/rfc822 text/plain message/rfc822 text/plain message/rfc822 text/plain message/rfc822 text/plain text/plain .. testcleanup:: somefile.close() Optional *fp* is a file-like object to print the output to. It must be suitable for Python's :func:`print` function. *level* is used internally. *include_default*, if true, prints the default type as well. PK 3]2library/fractions.rst.txtnu[:mod:`fractions` --- Rational numbers ===================================== .. module:: fractions :synopsis: Rational numbers. .. moduleauthor:: Jeffrey Yasskin .. sectionauthor:: Jeffrey Yasskin **Source code:** :source:`Lib/fractions.py` -------------- The :mod:`fractions` module provides support for rational number arithmetic. A Fraction instance can be constructed from a pair of integers, from another rational number, or from a string. .. class:: Fraction(numerator=0, denominator=1) Fraction(other_fraction) Fraction(float) Fraction(decimal) Fraction(string) The first version requires that *numerator* and *denominator* are instances of :class:`numbers.Rational` and returns a new :class:`Fraction` instance with value ``numerator/denominator``. If *denominator* is :const:`0`, it raises a :exc:`ZeroDivisionError`. The second version requires that *other_fraction* is an instance of :class:`numbers.Rational` and returns a :class:`Fraction` instance with the same value. The next two versions accept either a :class:`float` or a :class:`decimal.Decimal` instance, and return a :class:`Fraction` instance with exactly the same value. Note that due to the usual issues with binary floating-point (see :ref:`tut-fp-issues`), the argument to ``Fraction(1.1)`` is not exactly equal to 11/10, and so ``Fraction(1.1)`` does *not* return ``Fraction(11, 10)`` as one might expect. (But see the documentation for the :meth:`limit_denominator` method below.) The last version of the constructor expects a string or unicode instance. The usual form for this instance is:: [sign] numerator ['/' denominator] where the optional ``sign`` may be either '+' or '-' and ``numerator`` and ``denominator`` (if present) are strings of decimal digits. In addition, any string that represents a finite value and is accepted by the :class:`float` constructor is also accepted by the :class:`Fraction` constructor. In either form the input string may also have leading and/or trailing whitespace. Here are some examples:: >>> from fractions import Fraction >>> Fraction(16, -10) Fraction(-8, 5) >>> Fraction(123) Fraction(123, 1) >>> Fraction() Fraction(0, 1) >>> Fraction('3/7') Fraction(3, 7) >>> Fraction(' -3/7 ') Fraction(-3, 7) >>> Fraction('1.414213 \t\n') Fraction(1414213, 1000000) >>> Fraction('-.125') Fraction(-1, 8) >>> Fraction('7e-6') Fraction(7, 1000000) >>> Fraction(2.25) Fraction(9, 4) >>> Fraction(1.1) Fraction(2476979795053773, 2251799813685248) >>> from decimal import Decimal >>> Fraction(Decimal('1.1')) Fraction(11, 10) The :class:`Fraction` class inherits from the abstract base class :class:`numbers.Rational`, and implements all of the methods and operations from that class. :class:`Fraction` instances are hashable, and should be treated as immutable. In addition, :class:`Fraction` has the following properties and methods: .. versionchanged:: 3.2 The :class:`Fraction` constructor now accepts :class:`float` and :class:`decimal.Decimal` instances. .. attribute:: numerator Numerator of the Fraction in lowest term. .. attribute:: denominator Denominator of the Fraction in lowest term. .. method:: from_float(flt) This class method constructs a :class:`Fraction` representing the exact value of *flt*, which must be a :class:`float`. Beware that ``Fraction.from_float(0.3)`` is not the same value as ``Fraction(3, 10)``. .. note:: From Python 3.2 onwards, you can also construct a :class:`Fraction` instance directly from a :class:`float`. .. method:: from_decimal(dec) This class method constructs a :class:`Fraction` representing the exact value of *dec*, which must be a :class:`decimal.Decimal` instance. .. note:: From Python 3.2 onwards, you can also construct a :class:`Fraction` instance directly from a :class:`decimal.Decimal` instance. .. method:: limit_denominator(max_denominator=1000000) Finds and returns the closest :class:`Fraction` to ``self`` that has denominator at most max_denominator. This method is useful for finding rational approximations to a given floating-point number: >>> from fractions import Fraction >>> Fraction('3.1415926535897932').limit_denominator(1000) Fraction(355, 113) or for recovering a rational number that's represented as a float: >>> from math import pi, cos >>> Fraction(cos(pi/3)) Fraction(4503599627370497, 9007199254740992) >>> Fraction(cos(pi/3)).limit_denominator() Fraction(1, 2) >>> Fraction(1.1).limit_denominator() Fraction(11, 10) .. method:: __floor__() Returns the greatest :class:`int` ``<= self``. This method can also be accessed through the :func:`math.floor` function: >>> from math import floor >>> floor(Fraction(355, 113)) 3 .. method:: __ceil__() Returns the least :class:`int` ``>= self``. This method can also be accessed through the :func:`math.ceil` function. .. method:: __round__() __round__(ndigits) The first version returns the nearest :class:`int` to ``self``, rounding half to even. The second version rounds ``self`` to the nearest multiple of ``Fraction(1, 10**ndigits)`` (logically, if ``ndigits`` is negative), again rounding half toward even. This method can also be accessed through the :func:`round` function. .. function:: gcd(a, b) Return the greatest common divisor of the integers *a* and *b*. If either *a* or *b* is nonzero, then the absolute value of ``gcd(a, b)`` is the largest integer that divides both *a* and *b*. ``gcd(a,b)`` has the same sign as *b* if *b* is nonzero; otherwise it takes the sign of *a*. ``gcd(0, 0)`` returns ``0``. .. deprecated:: 3.5 Use :func:`math.gcd` instead. .. seealso:: Module :mod:`numbers` The abstract base classes making up the numeric tower. PK 3]}tlibrary/concurrency.rst.txtnu[.. _concurrency: ******************** Concurrent Execution ******************** The modules described in this chapter provide support for concurrent execution of code. The appropriate choice of tool will depend on the task to be executed (CPU bound vs IO bound) and preferred style of development (event driven cooperative multitasking vs preemptive multitasking). Here's an overview: .. toctree:: threading.rst multiprocessing.rst concurrent.rst concurrent.futures.rst subprocess.rst sched.rst queue.rst The following are support modules for some of the above services: .. toctree:: dummy_threading.rst _thread.rst _dummy_thread.rst PK 3]?e  library/fnmatch.rst.txtnu[:mod:`fnmatch` --- Unix filename pattern matching ================================================= .. module:: fnmatch :synopsis: Unix shell style filename pattern matching. **Source code:** :source:`Lib/fnmatch.py` .. index:: single: filenames; wildcard expansion .. index:: module: re -------------- This module provides support for Unix shell-style wildcards, which are *not* the same as regular expressions (which are documented in the :mod:`re` module). The special characters used in shell-style wildcards are: +------------+------------------------------------+ | Pattern | Meaning | +============+====================================+ | ``*`` | matches everything | +------------+------------------------------------+ | ``?`` | matches any single character | +------------+------------------------------------+ | ``[seq]`` | matches any character in *seq* | +------------+------------------------------------+ | ``[!seq]`` | matches any character not in *seq* | +------------+------------------------------------+ For a literal match, wrap the meta-characters in brackets. For example, ``'[?]'`` matches the character ``'?'``. .. index:: module: glob Note that the filename separator (``'/'`` on Unix) is *not* special to this module. See module :mod:`glob` for pathname expansion (:mod:`glob` uses :func:`fnmatch` to match pathname segments). Similarly, filenames starting with a period are not special for this module, and are matched by the ``*`` and ``?`` patterns. .. function:: fnmatch(filename, pattern) Test whether the *filename* string matches the *pattern* string, returning :const:`True` or :const:`False`. Both parameters are case-normalized using :func:`os.path.normcase`. :func:`fnmatchcase` can be used to perform a case-sensitive comparison, regardless of whether that's standard for the operating system. This example will print all file names in the current directory with the extension ``.txt``:: import fnmatch import os for file in os.listdir('.'): if fnmatch.fnmatch(file, '*.txt'): print(file) .. function:: fnmatchcase(filename, pattern) Test whether *filename* matches *pattern*, returning :const:`True` or :const:`False`; the comparison is case-sensitive and does not apply :func:`os.path.normcase`. .. function:: filter(names, pattern) Return the subset of the list of *names* that match *pattern*. It is the same as ``[n for n in names if fnmatch(n, pattern)]``, but implemented more efficiently. .. function:: translate(pattern) Return the shell-style *pattern* converted to a regular expression for using with :func:`re.match`. Example: >>> import fnmatch, re >>> >>> regex = fnmatch.translate('*.txt') >>> regex '(?s:.*\\.txt)\\Z' >>> reobj = re.compile(regex) >>> reobj.match('foobar.txt') <_sre.SRE_Match object; span=(0, 10), match='foobar.txt'> .. seealso:: Module :mod:`glob` Unix shell-style path expansion. PK 3]flibrary/functions.rst.txtnu[.. XXX document all delegations to __special__ methods .. _built-in-funcs: Built-in Functions ================== The Python interpreter has a number of functions and types built into it that are always available. They are listed here in alphabetical order. =================== ================= ================== ================ ==================== .. .. Built-in Functions .. .. =================== ================= ================== ================ ==================== :func:`abs` |func-dict|_ :func:`help` :func:`min` :func:`setattr` :func:`all` :func:`dir` :func:`hex` :func:`next` :func:`slice` :func:`any` :func:`divmod` :func:`id` :func:`object` :func:`sorted` :func:`ascii` :func:`enumerate` :func:`input` :func:`oct` :func:`staticmethod` :func:`bin` :func:`eval` :func:`int` :func:`open` |func-str|_ :func:`bool` :func:`exec` :func:`isinstance` :func:`ord` :func:`sum` |func-bytearray|_ :func:`filter` :func:`issubclass` :func:`pow` :func:`super` |func-bytes|_ :func:`float` :func:`iter` :func:`print` |func-tuple|_ :func:`callable` :func:`format` :func:`len` :func:`property` :func:`type` :func:`chr` |func-frozenset|_ |func-list|_ |func-range|_ :func:`vars` :func:`classmethod` :func:`getattr` :func:`locals` :func:`repr` :func:`zip` :func:`compile` :func:`globals` :func:`map` :func:`reversed` :func:`__import__` :func:`complex` :func:`hasattr` :func:`max` :func:`round` :func:`delattr` :func:`hash` |func-memoryview|_ |func-set|_ =================== ================= ================== ================ ==================== .. using :func:`dict` would create a link to another page, so local targets are used, with replacement texts to make the output in the table consistent .. |func-dict| replace:: ``dict()`` .. |func-frozenset| replace:: ``frozenset()`` .. |func-memoryview| replace:: ``memoryview()`` .. |func-set| replace:: ``set()`` .. |func-list| replace:: ``list()`` .. |func-str| replace:: ``str()`` .. |func-tuple| replace:: ``tuple()`` .. |func-range| replace:: ``range()`` .. |func-bytearray| replace:: ``bytearray()`` .. |func-bytes| replace:: ``bytes()`` .. function:: abs(x) Return the absolute value of a number. The argument may be an integer or a floating point number. If the argument is a complex number, its magnitude is returned. .. function:: all(iterable) Return ``True`` if all elements of the *iterable* are true (or if the iterable is empty). Equivalent to:: def all(iterable): for element in iterable: if not element: return False return True .. function:: any(iterable) Return ``True`` if any element of the *iterable* is true. If the iterable is empty, return ``False``. Equivalent to:: def any(iterable): for element in iterable: if element: return True return False .. function:: ascii(object) As :func:`repr`, return a string containing a printable representation of an object, but escape the non-ASCII characters in the string returned by :func:`repr` using ``\x``, ``\u`` or ``\U`` escapes. This generates a string similar to that returned by :func:`repr` in Python 2. .. function:: bin(x) Convert an integer number to a binary string prefixed with "0b". The result is a valid Python expression. If *x* is not a Python :class:`int` object, it has to define an :meth:`__index__` method that returns an integer. Some examples: >>> bin(3) '0b11' >>> bin(-10) '-0b1010' If prefix "0b" is desired or not, you can use either of the following ways. >>> format(14, '#b'), format(14, 'b') ('0b1110', '1110') >>> f'{14:#b}', f'{14:b}' ('0b1110', '1110') See also :func:`format` for more information. .. class:: bool([x]) Return a Boolean value, i.e. one of ``True`` or ``False``. *x* is converted using the standard :ref:`truth testing procedure `. If *x* is false or omitted, this returns ``False``; otherwise it returns ``True``. The :class:`bool` class is a subclass of :class:`int` (see :ref:`typesnumeric`). It cannot be subclassed further. Its only instances are ``False`` and ``True`` (see :ref:`bltin-boolean-values`). .. index:: pair: Boolean; type .. _func-bytearray: .. class:: bytearray([source[, encoding[, errors]]]) :noindex: Return a new array of bytes. The :class:`bytearray` class is a mutable sequence of integers in the range 0 <= x < 256. It has most of the usual methods of mutable sequences, described in :ref:`typesseq-mutable`, as well as most methods that the :class:`bytes` type has, see :ref:`bytes-methods`. The optional *source* parameter can be used to initialize the array in a few different ways: * If it is a *string*, you must also give the *encoding* (and optionally, *errors*) parameters; :func:`bytearray` then converts the string to bytes using :meth:`str.encode`. * If it is an *integer*, the array will have that size and will be initialized with null bytes. * If it is an object conforming to the *buffer* interface, a read-only buffer of the object will be used to initialize the bytes array. * If it is an *iterable*, it must be an iterable of integers in the range ``0 <= x < 256``, which are used as the initial contents of the array. Without an argument, an array of size 0 is created. See also :ref:`binaryseq` and :ref:`typebytearray`. .. _func-bytes: .. class:: bytes([source[, encoding[, errors]]]) :noindex: Return a new "bytes" object, which is an immutable sequence of integers in the range ``0 <= x < 256``. :class:`bytes` is an immutable version of :class:`bytearray` -- it has the same non-mutating methods and the same indexing and slicing behavior. Accordingly, constructor arguments are interpreted as for :func:`bytearray`. Bytes objects can also be created with literals, see :ref:`strings`. See also :ref:`binaryseq`, :ref:`typebytes`, and :ref:`bytes-methods`. .. function:: callable(object) Return :const:`True` if the *object* argument appears callable, :const:`False` if not. If this returns true, it is still possible that a call fails, but if it is false, calling *object* will never succeed. Note that classes are callable (calling a class returns a new instance); instances are callable if their class has a :meth:`__call__` method. .. versionadded:: 3.2 This function was first removed in Python 3.0 and then brought back in Python 3.2. .. function:: chr(i) Return the string representing a character whose Unicode code point is the integer *i*. For example, ``chr(97)`` returns the string ``'a'``, while ``chr(8364)`` returns the string ``'€'``. This is the inverse of :func:`ord`. The valid range for the argument is from 0 through 1,114,111 (0x10FFFF in base 16). :exc:`ValueError` will be raised if *i* is outside that range. .. decorator:: classmethod Transform a method into a class method. A class method receives the class as implicit first argument, just like an instance method receives the instance. To declare a class method, use this idiom:: class C: @classmethod def f(cls, arg1, arg2, ...): ... The ``@classmethod`` form is a function :term:`decorator` -- see the description of function definitions in :ref:`function` for details. It can be called either on the class (such as ``C.f()``) or on an instance (such as ``C().f()``). The instance is ignored except for its class. If a class method is called for a derived class, the derived class object is passed as the implied first argument. Class methods are different than C++ or Java static methods. If you want those, see :func:`staticmethod` in this section. For more information on class methods, consult the documentation on the standard type hierarchy in :ref:`types`. .. function:: compile(source, filename, mode, flags=0, dont_inherit=False, optimize=-1) Compile the *source* into a code or AST object. Code objects can be executed by :func:`exec` or :func:`eval`. *source* can either be a normal string, a byte string, or an AST object. Refer to the :mod:`ast` module documentation for information on how to work with AST objects. The *filename* argument should give the file from which the code was read; pass some recognizable value if it wasn't read from a file (``''`` is commonly used). The *mode* argument specifies what kind of code must be compiled; it can be ``'exec'`` if *source* consists of a sequence of statements, ``'eval'`` if it consists of a single expression, or ``'single'`` if it consists of a single interactive statement (in the latter case, expression statements that evaluate to something other than ``None`` will be printed). The optional arguments *flags* and *dont_inherit* control which :ref:`future statements ` affect the compilation of *source*. If neither is present (or both are zero) the code is compiled with those future statements that are in effect in the code that is calling :func:`compile`. If the *flags* argument is given and *dont_inherit* is not (or is zero) then the future statements specified by the *flags* argument are used in addition to those that would be used anyway. If *dont_inherit* is a non-zero integer then the *flags* argument is it -- the future statements in effect around the call to compile are ignored. Future statements are specified by bits which can be bitwise ORed together to specify multiple statements. The bitfield required to specify a given feature can be found as the :attr:`~__future__._Feature.compiler_flag` attribute on the :class:`~__future__._Feature` instance in the :mod:`__future__` module. The argument *optimize* specifies the optimization level of the compiler; the default value of ``-1`` selects the optimization level of the interpreter as given by :option:`-O` options. Explicit levels are ``0`` (no optimization; ``__debug__`` is true), ``1`` (asserts are removed, ``__debug__`` is false) or ``2`` (docstrings are removed too). This function raises :exc:`SyntaxError` if the compiled source is invalid, and :exc:`ValueError` if the source contains null bytes. If you want to parse Python code into its AST representation, see :func:`ast.parse`. .. note:: When compiling a string with multi-line code in ``'single'`` or ``'eval'`` mode, input must be terminated by at least one newline character. This is to facilitate detection of incomplete and complete statements in the :mod:`code` module. .. warning:: It is possible to crash the Python interpreter with a sufficiently large/complex string when compiling to an AST object due to stack depth limitations in Python's AST compiler. .. versionchanged:: 3.2 Allowed use of Windows and Mac newlines. Also input in ``'exec'`` mode does not have to end in a newline anymore. Added the *optimize* parameter. .. versionchanged:: 3.5 Previously, :exc:`TypeError` was raised when null bytes were encountered in *source*. .. class:: complex([real[, imag]]) Return a complex number with the value *real* + *imag*\*1j or convert a string or number to a complex number. If the first parameter is a string, it will be interpreted as a complex number and the function must be called without a second parameter. The second parameter can never be a string. Each argument may be any numeric type (including complex). If *imag* is omitted, it defaults to zero and the constructor serves as a numeric conversion like :class:`int` and :class:`float`. If both arguments are omitted, returns ``0j``. .. note:: When converting from a string, the string must not contain whitespace around the central ``+`` or ``-`` operator. For example, ``complex('1+2j')`` is fine, but ``complex('1 + 2j')`` raises :exc:`ValueError`. The complex type is described in :ref:`typesnumeric`. .. versionchanged:: 3.6 Grouping digits with underscores as in code literals is allowed. .. function:: delattr(object, name) This is a relative of :func:`setattr`. The arguments are an object and a string. The string must be the name of one of the object's attributes. The function deletes the named attribute, provided the object allows it. For example, ``delattr(x, 'foobar')`` is equivalent to ``del x.foobar``. .. _func-dict: .. class:: dict(**kwarg) dict(mapping, **kwarg) dict(iterable, **kwarg) :noindex: Create a new dictionary. The :class:`dict` object is the dictionary class. See :class:`dict` and :ref:`typesmapping` for documentation about this class. For other containers see the built-in :class:`list`, :class:`set`, and :class:`tuple` classes, as well as the :mod:`collections` module. .. function:: dir([object]) Without arguments, return the list of names in the current local scope. With an argument, attempt to return a list of valid attributes for that object. If the object has a method named :meth:`__dir__`, this method will be called and must return the list of attributes. This allows objects that implement a custom :func:`__getattr__` or :func:`__getattribute__` function to customize the way :func:`dir` reports their attributes. If the object does not provide :meth:`__dir__`, the function tries its best to gather information from the object's :attr:`~object.__dict__` attribute, if defined, and from its type object. The resulting list is not necessarily complete, and may be inaccurate when the object has a custom :func:`__getattr__`. The default :func:`dir` mechanism behaves differently with different types of objects, as it attempts to produce the most relevant, rather than complete, information: * If the object is a module object, the list contains the names of the module's attributes. * If the object is a type or class object, the list contains the names of its attributes, and recursively of the attributes of its bases. * Otherwise, the list contains the object's attributes' names, the names of its class's attributes, and recursively of the attributes of its class's base classes. The resulting list is sorted alphabetically. For example: >>> import struct >>> dir() # show the names in the module namespace ['__builtins__', '__name__', 'struct'] >>> dir(struct) # show the names in the struct module # doctest: +SKIP ['Struct', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__initializing__', '__loader__', '__name__', '__package__', '_clearcache', 'calcsize', 'error', 'pack', 'pack_into', 'unpack', 'unpack_from'] >>> class Shape: ... def __dir__(self): ... return ['area', 'perimeter', 'location'] >>> s = Shape() >>> dir(s) ['area', 'location', 'perimeter'] .. note:: Because :func:`dir` is supplied primarily as a convenience for use at an interactive prompt, it tries to supply an interesting set of names more than it tries to supply a rigorously or consistently defined set of names, and its detailed behavior may change across releases. For example, metaclass attributes are not in the result list when the argument is a class. .. function:: divmod(a, b) Take two (non complex) numbers as arguments and return a pair of numbers consisting of their quotient and remainder when using integer division. With mixed operand types, the rules for binary arithmetic operators apply. For integers, the result is the same as ``(a // b, a % b)``. For floating point numbers the result is ``(q, a % b)``, where *q* is usually ``math.floor(a / b)`` but may be 1 less than that. In any case ``q * b + a % b`` is very close to *a*, if ``a % b`` is non-zero it has the same sign as *b*, and ``0 <= abs(a % b) < abs(b)``. .. function:: enumerate(iterable, start=0) Return an enumerate object. *iterable* must be a sequence, an :term:`iterator`, or some other object which supports iteration. The :meth:`~iterator.__next__` method of the iterator returned by :func:`enumerate` returns a tuple containing a count (from *start* which defaults to 0) and the values obtained from iterating over *iterable*. >>> seasons = ['Spring', 'Summer', 'Fall', 'Winter'] >>> list(enumerate(seasons)) [(0, 'Spring'), (1, 'Summer'), (2, 'Fall'), (3, 'Winter')] >>> list(enumerate(seasons, start=1)) [(1, 'Spring'), (2, 'Summer'), (3, 'Fall'), (4, 'Winter')] Equivalent to:: def enumerate(sequence, start=0): n = start for elem in sequence: yield n, elem n += 1 .. function:: eval(expression, globals=None, locals=None) The arguments are a string and optional globals and locals. If provided, *globals* must be a dictionary. If provided, *locals* can be any mapping object. The *expression* argument is parsed and evaluated as a Python expression (technically speaking, a condition list) using the *globals* and *locals* dictionaries as global and local namespace. If the *globals* dictionary is present and does not contain a value for the key ``__builtins__``, a reference to the dictionary of the built-in module :mod:`builtins` is inserted under that key before *expression* is parsed. This means that *expression* normally has full access to the standard :mod:`builtins` module and restricted environments are propagated. If the *locals* dictionary is omitted it defaults to the *globals* dictionary. If both dictionaries are omitted, the expression is executed in the environment where :func:`eval` is called. The return value is the result of the evaluated expression. Syntax errors are reported as exceptions. Example: >>> x = 1 >>> eval('x+1') 2 This function can also be used to execute arbitrary code objects (such as those created by :func:`compile`). In this case pass a code object instead of a string. If the code object has been compiled with ``'exec'`` as the *mode* argument, :func:`eval`\'s return value will be ``None``. Hints: dynamic execution of statements is supported by the :func:`exec` function. The :func:`globals` and :func:`locals` functions returns the current global and local dictionary, respectively, which may be useful to pass around for use by :func:`eval` or :func:`exec`. See :func:`ast.literal_eval` for a function that can safely evaluate strings with expressions containing only literals. .. index:: builtin: exec .. function:: exec(object[, globals[, locals]]) This function supports dynamic execution of Python code. *object* must be either a string or a code object. If it is a string, the string is parsed as a suite of Python statements which is then executed (unless a syntax error occurs). [#]_ If it is a code object, it is simply executed. In all cases, the code that's executed is expected to be valid as file input (see the section "File input" in the Reference Manual). Be aware that the :keyword:`return` and :keyword:`yield` statements may not be used outside of function definitions even within the context of code passed to the :func:`exec` function. The return value is ``None``. In all cases, if the optional parts are omitted, the code is executed in the current scope. If only *globals* is provided, it must be a dictionary, which will be used for both the global and the local variables. If *globals* and *locals* are given, they are used for the global and local variables, respectively. If provided, *locals* can be any mapping object. Remember that at module level, globals and locals are the same dictionary. If exec gets two separate objects as *globals* and *locals*, the code will be executed as if it were embedded in a class definition. If the *globals* dictionary does not contain a value for the key ``__builtins__``, a reference to the dictionary of the built-in module :mod:`builtins` is inserted under that key. That way you can control what builtins are available to the executed code by inserting your own ``__builtins__`` dictionary into *globals* before passing it to :func:`exec`. .. note:: The built-in functions :func:`globals` and :func:`locals` return the current global and local dictionary, respectively, which may be useful to pass around for use as the second and third argument to :func:`exec`. .. note:: The default *locals* act as described for function :func:`locals` below: modifications to the default *locals* dictionary should not be attempted. Pass an explicit *locals* dictionary if you need to see effects of the code on *locals* after function :func:`exec` returns. .. function:: filter(function, iterable) Construct an iterator from those elements of *iterable* for which *function* returns true. *iterable* may be either a sequence, a container which supports iteration, or an iterator. If *function* is ``None``, the identity function is assumed, that is, all elements of *iterable* that are false are removed. Note that ``filter(function, iterable)`` is equivalent to the generator expression ``(item for item in iterable if function(item))`` if function is not ``None`` and ``(item for item in iterable if item)`` if function is ``None``. See :func:`itertools.filterfalse` for the complementary function that returns elements of *iterable* for which *function* returns false. .. class:: float([x]) .. index:: single: NaN single: Infinity Return a floating point number constructed from a number or string *x*. If the argument is a string, it should contain a decimal number, optionally preceded by a sign, and optionally embedded in whitespace. The optional sign may be ``'+'`` or ``'-'``; a ``'+'`` sign has no effect on the value produced. The argument may also be a string representing a NaN (not-a-number), or a positive or negative infinity. More precisely, the input must conform to the following grammar after leading and trailing whitespace characters are removed: .. productionlist:: sign: "+" | "-" infinity: "Infinity" | "inf" nan: "nan" numeric_value: `floatnumber` | `infinity` | `nan` numeric_string: [`sign`] `numeric_value` Here ``floatnumber`` is the form of a Python floating-point literal, described in :ref:`floating`. Case is not significant, so, for example, "inf", "Inf", "INFINITY" and "iNfINity" are all acceptable spellings for positive infinity. Otherwise, if the argument is an integer or a floating point number, a floating point number with the same value (within Python's floating point precision) is returned. If the argument is outside the range of a Python float, an :exc:`OverflowError` will be raised. For a general Python object ``x``, ``float(x)`` delegates to ``x.__float__()``. If no argument is given, ``0.0`` is returned. Examples:: >>> float('+1.23') 1.23 >>> float(' -12345\n') -12345.0 >>> float('1e-003') 0.001 >>> float('+1E6') 1000000.0 >>> float('-Infinity') -inf The float type is described in :ref:`typesnumeric`. .. versionchanged:: 3.6 Grouping digits with underscores as in code literals is allowed. .. index:: single: __format__ single: string; format() (built-in function) .. function:: format(value[, format_spec]) Convert a *value* to a "formatted" representation, as controlled by *format_spec*. The interpretation of *format_spec* will depend on the type of the *value* argument, however there is a standard formatting syntax that is used by most built-in types: :ref:`formatspec`. The default *format_spec* is an empty string which usually gives the same effect as calling :func:`str(value) `. A call to ``format(value, format_spec)`` is translated to ``type(value).__format__(value, format_spec)`` which bypasses the instance dictionary when searching for the value's :meth:`__format__` method. A :exc:`TypeError` exception is raised if the method search reaches :mod:`object` and the *format_spec* is non-empty, or if either the *format_spec* or the return value are not strings. .. versionchanged:: 3.4 ``object().__format__(format_spec)`` raises :exc:`TypeError` if *format_spec* is not an empty string. .. _func-frozenset: .. class:: frozenset([iterable]) :noindex: Return a new :class:`frozenset` object, optionally with elements taken from *iterable*. ``frozenset`` is a built-in class. See :class:`frozenset` and :ref:`types-set` for documentation about this class. For other containers see the built-in :class:`set`, :class:`list`, :class:`tuple`, and :class:`dict` classes, as well as the :mod:`collections` module. .. function:: getattr(object, name[, default]) Return the value of the named attribute of *object*. *name* must be a string. If the string is the name of one of the object's attributes, the result is the value of that attribute. For example, ``getattr(x, 'foobar')`` is equivalent to ``x.foobar``. If the named attribute does not exist, *default* is returned if provided, otherwise :exc:`AttributeError` is raised. .. function:: globals() Return a dictionary representing the current global symbol table. This is always the dictionary of the current module (inside a function or method, this is the module where it is defined, not the module from which it is called). .. function:: hasattr(object, name) The arguments are an object and a string. The result is ``True`` if the string is the name of one of the object's attributes, ``False`` if not. (This is implemented by calling ``getattr(object, name)`` and seeing whether it raises an :exc:`AttributeError` or not.) .. function:: hash(object) Return the hash value of the object (if it has one). Hash values are integers. They are used to quickly compare dictionary keys during a dictionary lookup. Numeric values that compare equal have the same hash value (even if they are of different types, as is the case for 1 and 1.0). .. note:: For objects with custom :meth:`__hash__` methods, note that :func:`hash` truncates the return value based on the bit width of the host machine. See :meth:`__hash__` for details. .. function:: help([object]) Invoke the built-in help system. (This function is intended for interactive use.) If no argument is given, the interactive help system starts on the interpreter console. If the argument is a string, then the string is looked up as the name of a module, function, class, method, keyword, or documentation topic, and a help page is printed on the console. If the argument is any other kind of object, a help page on the object is generated. This function is added to the built-in namespace by the :mod:`site` module. .. versionchanged:: 3.4 Changes to :mod:`pydoc` and :mod:`inspect` mean that the reported signatures for callables are now more comprehensive and consistent. .. function:: hex(x) Convert an integer number to a lowercase hexadecimal string prefixed with "0x". If *x* is not a Python :class:`int` object, it has to define an :meth:`__index__` method that returns an integer. Some examples: >>> hex(255) '0xff' >>> hex(-42) '-0x2a' If you want to convert an integer number to an uppercase or lower hexadecimal string with prefix or not, you can use either of the following ways: >>> '%#x' % 255, '%x' % 255, '%X' % 255 ('0xff', 'ff', 'FF') >>> format(255, '#x'), format(255, 'x'), format(255, 'X') ('0xff', 'ff', 'FF') >>> f'{255:#x}', f'{255:x}', f'{255:X}' ('0xff', 'ff', 'FF') See also :func:`format` for more information. See also :func:`int` for converting a hexadecimal string to an integer using a base of 16. .. note:: To obtain a hexadecimal string representation for a float, use the :meth:`float.hex` method. .. function:: id(object) Return the "identity" of an object. This is an integer which is guaranteed to be unique and constant for this object during its lifetime. Two objects with non-overlapping lifetimes may have the same :func:`id` value. .. impl-detail:: This is the address of the object in memory. .. function:: input([prompt]) If the *prompt* argument is present, it is written to standard output without a trailing newline. The function then reads a line from input, converts it to a string (stripping a trailing newline), and returns that. When EOF is read, :exc:`EOFError` is raised. Example:: >>> s = input('--> ') # doctest: +SKIP --> Monty Python's Flying Circus >>> s # doctest: +SKIP "Monty Python's Flying Circus" If the :mod:`readline` module was loaded, then :func:`input` will use it to provide elaborate line editing and history features. .. class:: int(x=0) int(x, base=10) Return an integer object constructed from a number or string *x*, or return ``0`` if no arguments are given. If *x* defines :meth:`__int__`, ``int(x)`` returns ``x.__int__()``. If *x* defines :meth:`__trunc__`, it returns ``x.__trunc__()``. For floating point numbers, this truncates towards zero. If *x* is not a number or if *base* is given, then *x* must be a string, :class:`bytes`, or :class:`bytearray` instance representing an :ref:`integer literal ` in radix *base*. Optionally, the literal can be preceded by ``+`` or ``-`` (with no space in between) and surrounded by whitespace. A base-n literal consists of the digits 0 to n-1, with ``a`` to ``z`` (or ``A`` to ``Z``) having values 10 to 35. The default *base* is 10. The allowed values are 0 and 2--36. Base-2, -8, and -16 literals can be optionally prefixed with ``0b``/``0B``, ``0o``/``0O``, or ``0x``/``0X``, as with integer literals in code. Base 0 means to interpret exactly as a code literal, so that the actual base is 2, 8, 10, or 16, and so that ``int('010', 0)`` is not legal, while ``int('010')`` is, as well as ``int('010', 8)``. The integer type is described in :ref:`typesnumeric`. .. versionchanged:: 3.4 If *base* is not an instance of :class:`int` and the *base* object has a :meth:`base.__index__ ` method, that method is called to obtain an integer for the base. Previous versions used :meth:`base.__int__ ` instead of :meth:`base.__index__ `. .. versionchanged:: 3.6 Grouping digits with underscores as in code literals is allowed. .. function:: isinstance(object, classinfo) Return true if the *object* argument is an instance of the *classinfo* argument, or of a (direct, indirect or :term:`virtual `) subclass thereof. If *object* is not an object of the given type, the function always returns false. If *classinfo* is a tuple of type objects (or recursively, other such tuples), return true if *object* is an instance of any of the types. If *classinfo* is not a type or tuple of types and such tuples, a :exc:`TypeError` exception is raised. .. function:: issubclass(class, classinfo) Return true if *class* is a subclass (direct, indirect or :term:`virtual `) of *classinfo*. A class is considered a subclass of itself. *classinfo* may be a tuple of class objects, in which case every entry in *classinfo* will be checked. In any other case, a :exc:`TypeError` exception is raised. .. function:: iter(object[, sentinel]) Return an :term:`iterator` object. The first argument is interpreted very differently depending on the presence of the second argument. Without a second argument, *object* must be a collection object which supports the iteration protocol (the :meth:`__iter__` method), or it must support the sequence protocol (the :meth:`__getitem__` method with integer arguments starting at ``0``). If it does not support either of those protocols, :exc:`TypeError` is raised. If the second argument, *sentinel*, is given, then *object* must be a callable object. The iterator created in this case will call *object* with no arguments for each call to its :meth:`~iterator.__next__` method; if the value returned is equal to *sentinel*, :exc:`StopIteration` will be raised, otherwise the value will be returned. See also :ref:`typeiter`. One useful application of the second form of :func:`iter` is to read lines of a file until a certain line is reached. The following example reads a file until the :meth:`~io.TextIOBase.readline` method returns an empty string:: with open('mydata.txt') as fp: for line in iter(fp.readline, ''): process_line(line) .. function:: len(s) Return the length (the number of items) of an object. The argument may be a sequence (such as a string, bytes, tuple, list, or range) or a collection (such as a dictionary, set, or frozen set). .. _func-list: .. class:: list([iterable]) :noindex: Rather than being a function, :class:`list` is actually a mutable sequence type, as documented in :ref:`typesseq-list` and :ref:`typesseq`. .. function:: locals() Update and return a dictionary representing the current local symbol table. Free variables are returned by :func:`locals` when it is called in function blocks, but not in class blocks. .. note:: The contents of this dictionary should not be modified; changes may not affect the values of local and free variables used by the interpreter. .. function:: map(function, iterable, ...) Return an iterator that applies *function* to every item of *iterable*, yielding the results. If additional *iterable* arguments are passed, *function* must take that many arguments and is applied to the items from all iterables in parallel. With multiple iterables, the iterator stops when the shortest iterable is exhausted. For cases where the function inputs are already arranged into argument tuples, see :func:`itertools.starmap`\. .. function:: max(iterable, *[, key, default]) max(arg1, arg2, *args[, key]) Return the largest item in an iterable or the largest of two or more arguments. If one positional argument is provided, it should be an :term:`iterable`. The largest item in the iterable is returned. If two or more positional arguments are provided, the largest of the positional arguments is returned. There are two optional keyword-only arguments. The *key* argument specifies a one-argument ordering function like that used for :meth:`list.sort`. The *default* argument specifies an object to return if the provided iterable is empty. If the iterable is empty and *default* is not provided, a :exc:`ValueError` is raised. If multiple items are maximal, the function returns the first one encountered. This is consistent with other sort-stability preserving tools such as ``sorted(iterable, key=keyfunc, reverse=True)[0]`` and ``heapq.nlargest(1, iterable, key=keyfunc)``. .. versionadded:: 3.4 The *default* keyword-only argument. .. _func-memoryview: .. function:: memoryview(obj) :noindex: Return a "memory view" object created from the given argument. See :ref:`typememoryview` for more information. .. function:: min(iterable, *[, key, default]) min(arg1, arg2, *args[, key]) Return the smallest item in an iterable or the smallest of two or more arguments. If one positional argument is provided, it should be an :term:`iterable`. The smallest item in the iterable is returned. If two or more positional arguments are provided, the smallest of the positional arguments is returned. There are two optional keyword-only arguments. The *key* argument specifies a one-argument ordering function like that used for :meth:`list.sort`. The *default* argument specifies an object to return if the provided iterable is empty. If the iterable is empty and *default* is not provided, a :exc:`ValueError` is raised. If multiple items are minimal, the function returns the first one encountered. This is consistent with other sort-stability preserving tools such as ``sorted(iterable, key=keyfunc)[0]`` and ``heapq.nsmallest(1, iterable, key=keyfunc)``. .. versionadded:: 3.4 The *default* keyword-only argument. .. function:: next(iterator[, default]) Retrieve the next item from the *iterator* by calling its :meth:`~iterator.__next__` method. If *default* is given, it is returned if the iterator is exhausted, otherwise :exc:`StopIteration` is raised. .. class:: object() Return a new featureless object. :class:`object` is a base for all classes. It has the methods that are common to all instances of Python classes. This function does not accept any arguments. .. note:: :class:`object` does *not* have a :attr:`~object.__dict__`, so you can't assign arbitrary attributes to an instance of the :class:`object` class. .. function:: oct(x) Convert an integer number to an octal string prefixed with "0o". The result is a valid Python expression. If *x* is not a Python :class:`int` object, it has to define an :meth:`__index__` method that returns an integer. For example: >>> oct(8) '0o10' >>> oct(-56) '-0o70' If you want to convert an integer number to octal string either with prefix "0o" or not, you can use either of the following ways. >>> '%#o' % 10, '%o' % 10 ('0o12', '12') >>> format(10, '#o'), format(10, 'o') ('0o12', '12') >>> f'{10:#o}', f'{10:o}' ('0o12', '12') See also :func:`format` for more information. .. index:: single: file object; open() built-in function .. function:: open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None) Open *file* and return a corresponding :term:`file object`. If the file cannot be opened, an :exc:`OSError` is raised. *file* is a :term:`path-like object` giving the pathname (absolute or relative to the current working directory) of the file to be opened or an integer file descriptor of the file to be wrapped. (If a file descriptor is given, it is closed when the returned I/O object is closed, unless *closefd* is set to ``False``.) *mode* is an optional string that specifies the mode in which the file is opened. It defaults to ``'r'`` which means open for reading in text mode. Other common values are ``'w'`` for writing (truncating the file if it already exists), ``'x'`` for exclusive creation and ``'a'`` for appending (which on *some* Unix systems, means that *all* writes append to the end of the file regardless of the current seek position). In text mode, if *encoding* is not specified the encoding used is platform dependent: ``locale.getpreferredencoding(False)`` is called to get the current locale encoding. (For reading and writing raw bytes use binary mode and leave *encoding* unspecified.) The available modes are: .. _filemodes: .. index:: pair: file; modes ========= =============================================================== Character Meaning ========= =============================================================== ``'r'`` open for reading (default) ``'w'`` open for writing, truncating the file first ``'x'`` open for exclusive creation, failing if the file already exists ``'a'`` open for writing, appending to the end of the file if it exists ``'b'`` binary mode ``'t'`` text mode (default) ``'+'`` open a disk file for updating (reading and writing) ``'U'`` :term:`universal newlines` mode (deprecated) ========= =============================================================== The default mode is ``'r'`` (open for reading text, synonym of ``'rt'``). For binary read-write access, the mode ``'w+b'`` opens and truncates the file to 0 bytes. ``'r+b'`` opens the file without truncation. As mentioned in the :ref:`io-overview`, Python distinguishes between binary and text I/O. Files opened in binary mode (including ``'b'`` in the *mode* argument) return contents as :class:`bytes` objects without any decoding. In text mode (the default, or when ``'t'`` is included in the *mode* argument), the contents of the file are returned as :class:`str`, the bytes having been first decoded using a platform-dependent encoding or using the specified *encoding* if given. .. note:: Python doesn't depend on the underlying operating system's notion of text files; all the processing is done by Python itself, and is therefore platform-independent. *buffering* is an optional integer used to set the buffering policy. Pass 0 to switch buffering off (only allowed in binary mode), 1 to select line buffering (only usable in text mode), and an integer > 1 to indicate the size in bytes of a fixed-size chunk buffer. When no *buffering* argument is given, the default buffering policy works as follows: * Binary files are buffered in fixed-size chunks; the size of the buffer is chosen using a heuristic trying to determine the underlying device's "block size" and falling back on :attr:`io.DEFAULT_BUFFER_SIZE`. On many systems, the buffer will typically be 4096 or 8192 bytes long. * "Interactive" text files (files for which :meth:`~io.IOBase.isatty` returns ``True``) use line buffering. Other text files use the policy described above for binary files. *encoding* is the name of the encoding used to decode or encode the file. This should only be used in text mode. The default encoding is platform dependent (whatever :func:`locale.getpreferredencoding` returns), but any :term:`text encoding` supported by Python can be used. See the :mod:`codecs` module for the list of supported encodings. *errors* is an optional string that specifies how encoding and decoding errors are to be handled—this cannot be used in binary mode. A variety of standard error handlers are available (listed under :ref:`error-handlers`), though any error handling name that has been registered with :func:`codecs.register_error` is also valid. The standard names include: * ``'strict'`` to raise a :exc:`ValueError` exception if there is an encoding error. The default value of ``None`` has the same effect. * ``'ignore'`` ignores errors. Note that ignoring encoding errors can lead to data loss. * ``'replace'`` causes a replacement marker (such as ``'?'``) to be inserted where there is malformed data. * ``'surrogateescape'`` will represent any incorrect bytes as code points in the Unicode Private Use Area ranging from U+DC80 to U+DCFF. These private code points will then be turned back into the same bytes when the ``surrogateescape`` error handler is used when writing data. This is useful for processing files in an unknown encoding. * ``'xmlcharrefreplace'`` is only supported when writing to a file. Characters not supported by the encoding are replaced with the appropriate XML character reference ``&#nnn;``. * ``'backslashreplace'`` replaces malformed data by Python's backslashed escape sequences. * ``'namereplace'`` (also only supported when writing) replaces unsupported characters with ``\N{...}`` escape sequences. .. index:: single: universal newlines; open() built-in function *newline* controls how :term:`universal newlines` mode works (it only applies to text mode). It can be ``None``, ``''``, ``'\n'``, ``'\r'``, and ``'\r\n'``. It works as follows: * When reading input from the stream, if *newline* is ``None``, universal newlines mode is enabled. Lines in the input can end in ``'\n'``, ``'\r'``, or ``'\r\n'``, and these are translated into ``'\n'`` before being returned to the caller. If it is ``''``, universal newlines mode is enabled, but line endings are returned to the caller untranslated. If it has any of the other legal values, input lines are only terminated by the given string, and the line ending is returned to the caller untranslated. * When writing output to the stream, if *newline* is ``None``, any ``'\n'`` characters written are translated to the system default line separator, :data:`os.linesep`. If *newline* is ``''`` or ``'\n'``, no translation takes place. If *newline* is any of the other legal values, any ``'\n'`` characters written are translated to the given string. If *closefd* is ``False`` and a file descriptor rather than a filename was given, the underlying file descriptor will be kept open when the file is closed. If a filename is given *closefd* must be ``True`` (the default) otherwise an error will be raised. A custom opener can be used by passing a callable as *opener*. The underlying file descriptor for the file object is then obtained by calling *opener* with (*file*, *flags*). *opener* must return an open file descriptor (passing :mod:`os.open` as *opener* results in functionality similar to passing ``None``). The newly created file is :ref:`non-inheritable `. The following example uses the :ref:`dir_fd ` parameter of the :func:`os.open` function to open a file relative to a given directory:: >>> import os >>> dir_fd = os.open('somedir', os.O_RDONLY) >>> def opener(path, flags): ... return os.open(path, flags, dir_fd=dir_fd) ... >>> with open('spamspam.txt', 'w', opener=opener) as f: ... print('This will be written to somedir/spamspam.txt', file=f) ... >>> os.close(dir_fd) # don't leak a file descriptor The type of :term:`file object` returned by the :func:`open` function depends on the mode. When :func:`open` is used to open a file in a text mode (``'w'``, ``'r'``, ``'wt'``, ``'rt'``, etc.), it returns a subclass of :class:`io.TextIOBase` (specifically :class:`io.TextIOWrapper`). When used to open a file in a binary mode with buffering, the returned class is a subclass of :class:`io.BufferedIOBase`. The exact class varies: in read binary mode, it returns an :class:`io.BufferedReader`; in write binary and append binary modes, it returns an :class:`io.BufferedWriter`, and in read/write mode, it returns an :class:`io.BufferedRandom`. When buffering is disabled, the raw stream, a subclass of :class:`io.RawIOBase`, :class:`io.FileIO`, is returned. .. index:: single: line-buffered I/O single: unbuffered I/O single: buffer size, I/O single: I/O control; buffering single: binary mode single: text mode module: sys See also the file handling modules, such as, :mod:`fileinput`, :mod:`io` (where :func:`open` is declared), :mod:`os`, :mod:`os.path`, :mod:`tempfile`, and :mod:`shutil`. .. versionchanged:: 3.3 * The *opener* parameter was added. * The ``'x'`` mode was added. * :exc:`IOError` used to be raised, it is now an alias of :exc:`OSError`. * :exc:`FileExistsError` is now raised if the file opened in exclusive creation mode (``'x'``) already exists. .. versionchanged:: 3.4 * The file is now non-inheritable. .. deprecated-removed:: 3.4 4.0 The ``'U'`` mode. .. versionchanged:: 3.5 * If the system call is interrupted and the signal handler does not raise an exception, the function now retries the system call instead of raising an :exc:`InterruptedError` exception (see :pep:`475` for the rationale). * The ``'namereplace'`` error handler was added. .. versionchanged:: 3.6 * Support added to accept objects implementing :class:`os.PathLike`. * On Windows, opening a console buffer may return a subclass of :class:`io.RawIOBase` other than :class:`io.FileIO`. .. function:: ord(c) Given a string representing one Unicode character, return an integer representing the Unicode code point of that character. For example, ``ord('a')`` returns the integer ``97`` and ``ord('€')`` (Euro sign) returns ``8364``. This is the inverse of :func:`chr`. .. function:: pow(x, y[, z]) Return *x* to the power *y*; if *z* is present, return *x* to the power *y*, modulo *z* (computed more efficiently than ``pow(x, y) % z``). The two-argument form ``pow(x, y)`` is equivalent to using the power operator: ``x**y``. The arguments must have numeric types. With mixed operand types, the coercion rules for binary arithmetic operators apply. For :class:`int` operands, the result has the same type as the operands (after coercion) unless the second argument is negative; in that case, all arguments are converted to float and a float result is delivered. For example, ``10**2`` returns ``100``, but ``10**-2`` returns ``0.01``. If the second argument is negative, the third argument must be omitted. If *z* is present, *x* and *y* must be of integer types, and *y* must be non-negative. .. function:: print(*objects, sep=' ', end='\\n', file=sys.stdout, flush=False) Print *objects* to the text stream *file*, separated by *sep* and followed by *end*. *sep*, *end*, *file* and *flush*, if present, must be given as keyword arguments. All non-keyword arguments are converted to strings like :func:`str` does and written to the stream, separated by *sep* and followed by *end*. Both *sep* and *end* must be strings; they can also be ``None``, which means to use the default values. If no *objects* are given, :func:`print` will just write *end*. The *file* argument must be an object with a ``write(string)`` method; if it is not present or ``None``, :data:`sys.stdout` will be used. Since printed arguments are converted to text strings, :func:`print` cannot be used with binary mode file objects. For these, use ``file.write(...)`` instead. Whether output is buffered is usually determined by *file*, but if the *flush* keyword argument is true, the stream is forcibly flushed. .. versionchanged:: 3.3 Added the *flush* keyword argument. .. class:: property(fget=None, fset=None, fdel=None, doc=None) Return a property attribute. *fget* is a function for getting an attribute value. *fset* is a function for setting an attribute value. *fdel* is a function for deleting an attribute value. And *doc* creates a docstring for the attribute. A typical use is to define a managed attribute ``x``:: class C: def __init__(self): self._x = None def getx(self): return self._x def setx(self, value): self._x = value def delx(self): del self._x x = property(getx, setx, delx, "I'm the 'x' property.") If *c* is an instance of *C*, ``c.x`` will invoke the getter, ``c.x = value`` will invoke the setter and ``del c.x`` the deleter. If given, *doc* will be the docstring of the property attribute. Otherwise, the property will copy *fget*'s docstring (if it exists). This makes it possible to create read-only properties easily using :func:`property` as a :term:`decorator`:: class Parrot: def __init__(self): self._voltage = 100000 @property def voltage(self): """Get the current voltage.""" return self._voltage The ``@property`` decorator turns the :meth:`voltage` method into a "getter" for a read-only attribute with the same name, and it sets the docstring for *voltage* to "Get the current voltage." A property object has :attr:`~property.getter`, :attr:`~property.setter`, and :attr:`~property.deleter` methods usable as decorators that create a copy of the property with the corresponding accessor function set to the decorated function. This is best explained with an example:: class C: def __init__(self): self._x = None @property def x(self): """I'm the 'x' property.""" return self._x @x.setter def x(self, value): self._x = value @x.deleter def x(self): del self._x This code is exactly equivalent to the first example. Be sure to give the additional functions the same name as the original property (``x`` in this case.) The returned property object also has the attributes ``fget``, ``fset``, and ``fdel`` corresponding to the constructor arguments. .. versionchanged:: 3.5 The docstrings of property objects are now writeable. .. _func-range: .. function:: range(stop) range(start, stop[, step]) :noindex: Rather than being a function, :class:`range` is actually an immutable sequence type, as documented in :ref:`typesseq-range` and :ref:`typesseq`. .. function:: repr(object) Return a string containing a printable representation of an object. For many types, this function makes an attempt to return a string that would yield an object with the same value when passed to :func:`eval`, otherwise the representation is a string enclosed in angle brackets that contains the name of the type of the object together with additional information often including the name and address of the object. A class can control what this function returns for its instances by defining a :meth:`__repr__` method. .. function:: reversed(seq) Return a reverse :term:`iterator`. *seq* must be an object which has a :meth:`__reversed__` method or supports the sequence protocol (the :meth:`__len__` method and the :meth:`__getitem__` method with integer arguments starting at ``0``). .. function:: round(number[, ndigits]) Return *number* rounded to *ndigits* precision after the decimal point. If *ndigits* is omitted or is ``None``, it returns the nearest integer to its input. For the built-in types supporting :func:`round`, values are rounded to the closest multiple of 10 to the power minus *ndigits*; if two multiples are equally close, rounding is done toward the even choice (so, for example, both ``round(0.5)`` and ``round(-0.5)`` are ``0``, and ``round(1.5)`` is ``2``). Any integer value is valid for *ndigits* (positive, zero, or negative). The return value is an integer if *ndigits* is omitted or ``None``. Otherwise the return value has the same type as *number*. For a general Python object ``number``, ``round`` delegates to ``number.__round__``. .. note:: The behavior of :func:`round` for floats can be surprising: for example, ``round(2.675, 2)`` gives ``2.67`` instead of the expected ``2.68``. This is not a bug: it's a result of the fact that most decimal fractions can't be represented exactly as a float. See :ref:`tut-fp-issues` for more information. .. _func-set: .. class:: set([iterable]) :noindex: Return a new :class:`set` object, optionally with elements taken from *iterable*. ``set`` is a built-in class. See :class:`set` and :ref:`types-set` for documentation about this class. For other containers see the built-in :class:`frozenset`, :class:`list`, :class:`tuple`, and :class:`dict` classes, as well as the :mod:`collections` module. .. function:: setattr(object, name, value) This is the counterpart of :func:`getattr`. The arguments are an object, a string and an arbitrary value. The string may name an existing attribute or a new attribute. The function assigns the value to the attribute, provided the object allows it. For example, ``setattr(x, 'foobar', 123)`` is equivalent to ``x.foobar = 123``. .. class:: slice(stop) slice(start, stop[, step]) .. index:: single: Numerical Python Return a :term:`slice` object representing the set of indices specified by ``range(start, stop, step)``. The *start* and *step* arguments default to ``None``. Slice objects have read-only data attributes :attr:`~slice.start`, :attr:`~slice.stop` and :attr:`~slice.step` which merely return the argument values (or their default). They have no other explicit functionality; however they are used by Numerical Python and other third party extensions. Slice objects are also generated when extended indexing syntax is used. For example: ``a[start:stop:step]`` or ``a[start:stop, i]``. See :func:`itertools.islice` for an alternate version that returns an iterator. .. function:: sorted(iterable, *, key=None, reverse=False) Return a new sorted list from the items in *iterable*. Has two optional arguments which must be specified as keyword arguments. *key* specifies a function of one argument that is used to extract a comparison key from each list element: ``key=str.lower``. The default value is ``None`` (compare the elements directly). *reverse* is a boolean value. If set to ``True``, then the list elements are sorted as if each comparison were reversed. Use :func:`functools.cmp_to_key` to convert an old-style *cmp* function to a *key* function. The built-in :func:`sorted` function is guaranteed to be stable. A sort is stable if it guarantees not to change the relative order of elements that compare equal --- this is helpful for sorting in multiple passes (for example, sort by department, then by salary grade). For sorting examples and a brief sorting tutorial, see :ref:`sortinghowto`. .. decorator:: staticmethod Transform a method into a static method. A static method does not receive an implicit first argument. To declare a static method, use this idiom:: class C: @staticmethod def f(arg1, arg2, ...): ... The ``@staticmethod`` form is a function :term:`decorator` -- see the description of function definitions in :ref:`function` for details. It can be called either on the class (such as ``C.f()``) or on an instance (such as ``C().f()``). The instance is ignored except for its class. Static methods in Python are similar to those found in Java or C++. Also see :func:`classmethod` for a variant that is useful for creating alternate class constructors. Like all decorators, it is also possible to call ``staticmethod`` as a regular function and do something with its result. This is needed in some cases where you need a reference to a function from a class body and you want to avoid the automatic transformation to instance method. For these cases, use this idiom:: class C: builtin_open = staticmethod(open) For more information on static methods, consult the documentation on the standard type hierarchy in :ref:`types`. .. index:: single: string; str() (built-in function) .. _func-str: .. class:: str(object='') str(object=b'', encoding='utf-8', errors='strict') :noindex: Return a :class:`str` version of *object*. See :func:`str` for details. ``str`` is the built-in string :term:`class`. For general information about strings, see :ref:`textseq`. .. function:: sum(iterable[, start]) Sums *start* and the items of an *iterable* from left to right and returns the total. *start* defaults to ``0``. The *iterable*'s items are normally numbers, and the start value is not allowed to be a string. For some use cases, there are good alternatives to :func:`sum`. The preferred, fast way to concatenate a sequence of strings is by calling ``''.join(sequence)``. To add floating point values with extended precision, see :func:`math.fsum`\. To concatenate a series of iterables, consider using :func:`itertools.chain`. .. function:: super([type[, object-or-type]]) Return a proxy object that delegates method calls to a parent or sibling class of *type*. This is useful for accessing inherited methods that have been overridden in a class. The search order is same as that used by :func:`getattr` except that the *type* itself is skipped. The :attr:`~class.__mro__` attribute of the *type* lists the method resolution search order used by both :func:`getattr` and :func:`super`. The attribute is dynamic and can change whenever the inheritance hierarchy is updated. If the second argument is omitted, the super object returned is unbound. If the second argument is an object, ``isinstance(obj, type)`` must be true. If the second argument is a type, ``issubclass(type2, type)`` must be true (this is useful for classmethods). There are two typical use cases for *super*. In a class hierarchy with single inheritance, *super* can be used to refer to parent classes without naming them explicitly, thus making the code more maintainable. This use closely parallels the use of *super* in other programming languages. The second use case is to support cooperative multiple inheritance in a dynamic execution environment. This use case is unique to Python and is not found in statically compiled languages or languages that only support single inheritance. This makes it possible to implement "diamond diagrams" where multiple base classes implement the same method. Good design dictates that this method have the same calling signature in every case (because the order of calls is determined at runtime, because that order adapts to changes in the class hierarchy, and because that order can include sibling classes that are unknown prior to runtime). For both use cases, a typical superclass call looks like this:: class C(B): def method(self, arg): super().method(arg) # This does the same thing as: # super(C, self).method(arg) Note that :func:`super` is implemented as part of the binding process for explicit dotted attribute lookups such as ``super().__getitem__(name)``. It does so by implementing its own :meth:`__getattribute__` method for searching classes in a predictable order that supports cooperative multiple inheritance. Accordingly, :func:`super` is undefined for implicit lookups using statements or operators such as ``super()[name]``. Also note that, aside from the zero argument form, :func:`super` is not limited to use inside methods. The two argument form specifies the arguments exactly and makes the appropriate references. The zero argument form only works inside a class definition, as the compiler fills in the necessary details to correctly retrieve the class being defined, as well as accessing the current instance for ordinary methods. For practical suggestions on how to design cooperative classes using :func:`super`, see `guide to using super() `_. .. _func-tuple: .. function:: tuple([iterable]) :noindex: Rather than being a function, :class:`tuple` is actually an immutable sequence type, as documented in :ref:`typesseq-tuple` and :ref:`typesseq`. .. class:: type(object) type(name, bases, dict) .. index:: object: type With one argument, return the type of an *object*. The return value is a type object and generally the same object as returned by :attr:`object.__class__ `. The :func:`isinstance` built-in function is recommended for testing the type of an object, because it takes subclasses into account. With three arguments, return a new type object. This is essentially a dynamic form of the :keyword:`class` statement. The *name* string is the class name and becomes the :attr:`~definition.__name__` attribute; the *bases* tuple itemizes the base classes and becomes the :attr:`~class.__bases__` attribute; and the *dict* dictionary is the namespace containing definitions for class body and is copied to a standard dictionary to become the :attr:`~object.__dict__` attribute. For example, the following two statements create identical :class:`type` objects: >>> class X: ... a = 1 ... >>> X = type('X', (object,), dict(a=1)) See also :ref:`bltin-type-objects`. .. versionchanged:: 3.6 Subclasses of :class:`type` which don't override ``type.__new__`` may no longer use the one-argument form to get the type of an object. .. function:: vars([object]) Return the :attr:`~object.__dict__` attribute for a module, class, instance, or any other object with a :attr:`~object.__dict__` attribute. Objects such as modules and instances have an updateable :attr:`~object.__dict__` attribute; however, other objects may have write restrictions on their :attr:`~object.__dict__` attributes (for example, classes use a :class:`types.MappingProxyType` to prevent direct dictionary updates). Without an argument, :func:`vars` acts like :func:`locals`. Note, the locals dictionary is only useful for reads since updates to the locals dictionary are ignored. .. function:: zip(*iterables) Make an iterator that aggregates elements from each of the iterables. Returns an iterator of tuples, where the *i*-th tuple contains the *i*-th element from each of the argument sequences or iterables. The iterator stops when the shortest input iterable is exhausted. With a single iterable argument, it returns an iterator of 1-tuples. With no arguments, it returns an empty iterator. Equivalent to:: def zip(*iterables): # zip('ABCD', 'xy') --> Ax By sentinel = object() iterators = [iter(it) for it in iterables] while iterators: result = [] for it in iterators: elem = next(it, sentinel) if elem is sentinel: return result.append(elem) yield tuple(result) The left-to-right evaluation order of the iterables is guaranteed. This makes possible an idiom for clustering a data series into n-length groups using ``zip(*[iter(s)]*n)``. This repeats the *same* iterator ``n`` times so that each output tuple has the result of ``n`` calls to the iterator. This has the effect of dividing the input into n-length chunks. :func:`zip` should only be used with unequal length inputs when you don't care about trailing, unmatched values from the longer iterables. If those values are important, use :func:`itertools.zip_longest` instead. :func:`zip` in conjunction with the ``*`` operator can be used to unzip a list:: >>> x = [1, 2, 3] >>> y = [4, 5, 6] >>> zipped = zip(x, y) >>> list(zipped) [(1, 4), (2, 5), (3, 6)] >>> x2, y2 = zip(*zip(x, y)) >>> x == list(x2) and y == list(y2) True .. function:: __import__(name, globals=None, locals=None, fromlist=(), level=0) .. index:: statement: import module: imp .. note:: This is an advanced function that is not needed in everyday Python programming, unlike :func:`importlib.import_module`. This function is invoked by the :keyword:`import` statement. It can be replaced (by importing the :mod:`builtins` module and assigning to ``builtins.__import__``) in order to change semantics of the :keyword:`import` statement, but doing so is **strongly** discouraged as it is usually simpler to use import hooks (see :pep:`302`) to attain the same goals and does not cause issues with code which assumes the default import implementation is in use. Direct use of :func:`__import__` is also discouraged in favor of :func:`importlib.import_module`. The function imports the module *name*, potentially using the given *globals* and *locals* to determine how to interpret the name in a package context. The *fromlist* gives the names of objects or submodules that should be imported from the module given by *name*. The standard implementation does not use its *locals* argument at all, and uses its *globals* only to determine the package context of the :keyword:`import` statement. *level* specifies whether to use absolute or relative imports. ``0`` (the default) means only perform absolute imports. Positive values for *level* indicate the number of parent directories to search relative to the directory of the module calling :func:`__import__` (see :pep:`328` for the details). When the *name* variable is of the form ``package.module``, normally, the top-level package (the name up till the first dot) is returned, *not* the module named by *name*. However, when a non-empty *fromlist* argument is given, the module named by *name* is returned. For example, the statement ``import spam`` results in bytecode resembling the following code:: spam = __import__('spam', globals(), locals(), [], 0) The statement ``import spam.ham`` results in this call:: spam = __import__('spam.ham', globals(), locals(), [], 0) Note how :func:`__import__` returns the toplevel module here because this is the object that is bound to a name by the :keyword:`import` statement. On the other hand, the statement ``from spam.ham import eggs, sausage as saus`` results in :: _temp = __import__('spam.ham', globals(), locals(), ['eggs', 'sausage'], 0) eggs = _temp.eggs saus = _temp.sausage Here, the ``spam.ham`` module is returned from :func:`__import__`. From this object, the names to import are retrieved and assigned to their respective names. If you simply want to import a module (potentially within a package) by name, use :func:`importlib.import_module`. .. versionchanged:: 3.3 Negative values for *level* are no longer supported (which also changes the default value to 0). .. rubric:: Footnotes .. [#] Note that the parser only accepts the Unix-style end of line convention. If you are reading the code from a file, make sure to use newline conversion mode to convert Windows or Mac-style newlines. PK 3]\C  library/py_compile.rst.txtnu[:mod:`py_compile` --- Compile Python source files ================================================= .. module:: py_compile :synopsis: Generate byte-code files from Python source files. .. sectionauthor:: Fred L. Drake, Jr. .. documentation based on module docstrings **Source code:** :source:`Lib/py_compile.py` .. index:: pair: file; byte-code -------------- The :mod:`py_compile` module provides a function to generate a byte-code file from a source file, and another function used when the module source file is invoked as a script. Though not often needed, this function can be useful when installing modules for shared use, especially if some of the users may not have permission to write the byte-code cache files in the directory containing the source code. .. exception:: PyCompileError Exception raised when an error occurs while attempting to compile the file. .. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1) Compile a source file to byte-code and write out the byte-code cache file. The source code is loaded from the file named *file*. The byte-code is written to *cfile*, which defaults to the :pep:`3147`/:pep:`488` path, ending in ``.pyc``. For example, if *file* is ``/foo/bar/baz.py`` *cfile* will default to ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. If *dfile* is specified, it is used as the name of the source file in error messages when instead of *file*. If *doraise* is true, a :exc:`PyCompileError` is raised when an error is encountered while compiling *file*. If *doraise* is false (the default), an error string is written to ``sys.stderr``, but no exception is raised. This function returns the path to byte-compiled file, i.e. whatever *cfile* value was used. If the path that *cfile* becomes (either explicitly specified or computed) is a symlink or non-regular file, :exc:`FileExistsError` will be raised. This is to act as a warning that import will turn those paths into regular files if it is allowed to write byte-compiled files to those paths. This is a side-effect of import using file renaming to place the final byte-compiled file into place to prevent concurrent file writing issues. *optimize* controls the optimization level and is passed to the built-in :func:`compile` function. The default of ``-1`` selects the optimization level of the current interpreter. .. versionchanged:: 3.2 Changed default value of *cfile* to be :PEP:`3147`-compliant. Previous default was *file* + ``'c'`` (``'o'`` if optimization was enabled). Also added the *optimize* parameter. .. versionchanged:: 3.4 Changed code to use :mod:`importlib` for the byte-code cache file writing. This means file creation/writing semantics now match what :mod:`importlib` does, e.g. permissions, write-and-move semantics, etc. Also added the caveat that :exc:`FileExistsError` is raised if *cfile* is a symlink or non-regular file. .. function:: main(args=None) Compile several source files. The files named in *args* (or on the command line, if *args* is ``None``) are compiled and the resulting byte-code is cached in the normal manner. This function does not search a directory structure to locate source files; it only compiles files named explicitly. If ``'-'`` is the only parameter in args, the list of files is taken from standard input. .. versionchanged:: 3.2 Added support for ``'-'``. When this module is run as a script, the :func:`main` is used to compile all the files named on the command line. The exit status is nonzero if one of the files could not be compiled. .. seealso:: Module :mod:`compileall` Utilities to compile all Python source files in a directory tree. PK 3]$2 2 library/copy.rst.txtnu[:mod:`copy` --- Shallow and deep copy operations ================================================ .. module:: copy :synopsis: Shallow and deep copy operations. **Source code:** :source:`Lib/copy.py` -------------- Assignment statements in Python do not copy objects, they create bindings between a target and an object. For collections that are mutable or contain mutable items, a copy is sometimes needed so one can change one copy without changing the other. This module provides generic shallow and deep copy operations (explained below). Interface summary: .. function:: copy(x) Return a shallow copy of *x*. .. function:: deepcopy(x) Return a deep copy of *x*. .. exception:: error Raised for module specific errors. The difference between shallow and deep copying is only relevant for compound objects (objects that contain other objects, like lists or class instances): * A *shallow copy* constructs a new compound object and then (to the extent possible) inserts *references* into it to the objects found in the original. * A *deep copy* constructs a new compound object and then, recursively, inserts *copies* into it of the objects found in the original. Two problems often exist with deep copy operations that don't exist with shallow copy operations: * Recursive objects (compound objects that, directly or indirectly, contain a reference to themselves) may cause a recursive loop. * Because deep copy copies everything it may copy too much, such as data which is intended to be shared between copies. The :func:`deepcopy` function avoids these problems by: * keeping a "memo" dictionary of objects already copied during the current copying pass; and * letting user-defined classes override the copying operation or the set of components copied. This module does not copy types like module, method, stack trace, stack frame, file, socket, window, array, or any similar types. It does "copy" functions and classes (shallow and deeply), by returning the original object unchanged; this is compatible with the way these are treated by the :mod:`pickle` module. Shallow copies of dictionaries can be made using :meth:`dict.copy`, and of lists by assigning a slice of the entire list, for example, ``copied_list = original_list[:]``. .. index:: module: pickle Classes can use the same interfaces to control copying that they use to control pickling. See the description of module :mod:`pickle` for information on these methods. In fact, the :mod:`copy` module uses the registered pickle functions from the :mod:`copyreg` module. .. index:: single: __copy__() (copy protocol) single: __deepcopy__() (copy protocol) In order for a class to define its own copy implementation, it can define special methods :meth:`__copy__` and :meth:`__deepcopy__`. The former is called to implement the shallow copy operation; no additional arguments are passed. The latter is called to implement the deep copy operation; it is passed one argument, the memo dictionary. If the :meth:`__deepcopy__` implementation needs to make a deep copy of a component, it should call the :func:`deepcopy` function with the component as first argument and the memo dictionary as second argument. .. seealso:: Module :mod:`pickle` Discussion of the special methods used to support object state retrieval and restoration. PK 3]Z"library/unicodedata.rst.txtnu[:mod:`unicodedata` --- Unicode Database ======================================= .. module:: unicodedata :synopsis: Access the Unicode Database. .. moduleauthor:: Marc-André Lemburg .. sectionauthor:: Marc-André Lemburg .. sectionauthor:: Martin v. Löwis .. index:: single: Unicode single: character pair: Unicode; database -------------- This module provides access to the Unicode Character Database (UCD) which defines character properties for all Unicode characters. The data contained in this database is compiled from the `UCD version 9.0.0 `_. The module uses the same names and symbols as defined by Unicode Standard Annex #44, `"Unicode Character Database" `_. It defines the following functions: .. function:: lookup(name) Look up character by name. If a character with the given name is found, return the corresponding character. If not found, :exc:`KeyError` is raised. .. versionchanged:: 3.3 Support for name aliases [#]_ and named sequences [#]_ has been added. .. function:: name(chr[, default]) Returns the name assigned to the character *chr* as a string. If no name is defined, *default* is returned, or, if not given, :exc:`ValueError` is raised. .. function:: decimal(chr[, default]) Returns the decimal value assigned to the character *chr* as integer. If no such value is defined, *default* is returned, or, if not given, :exc:`ValueError` is raised. .. function:: digit(chr[, default]) Returns the digit value assigned to the character *chr* as integer. If no such value is defined, *default* is returned, or, if not given, :exc:`ValueError` is raised. .. function:: numeric(chr[, default]) Returns the numeric value assigned to the character *chr* as float. If no such value is defined, *default* is returned, or, if not given, :exc:`ValueError` is raised. .. function:: category(chr) Returns the general category assigned to the character *chr* as string. .. function:: bidirectional(chr) Returns the bidirectional class assigned to the character *chr* as string. If no such value is defined, an empty string is returned. .. function:: combining(chr) Returns the canonical combining class assigned to the character *chr* as integer. Returns ``0`` if no combining class is defined. .. function:: east_asian_width(chr) Returns the east asian width assigned to the character *chr* as string. .. function:: mirrored(chr) Returns the mirrored property assigned to the character *chr* as integer. Returns ``1`` if the character has been identified as a "mirrored" character in bidirectional text, ``0`` otherwise. .. function:: decomposition(chr) Returns the character decomposition mapping assigned to the character *chr* as string. An empty string is returned in case no such mapping is defined. .. function:: normalize(form, unistr) Return the normal form *form* for the Unicode string *unistr*. Valid values for *form* are 'NFC', 'NFKC', 'NFD', and 'NFKD'. The Unicode standard defines various normalization forms of a Unicode string, based on the definition of canonical equivalence and compatibility equivalence. In Unicode, several characters can be expressed in various way. For example, the character U+00C7 (LATIN CAPITAL LETTER C WITH CEDILLA) can also be expressed as the sequence U+0043 (LATIN CAPITAL LETTER C) U+0327 (COMBINING CEDILLA). For each character, there are two normal forms: normal form C and normal form D. Normal form D (NFD) is also known as canonical decomposition, and translates each character into its decomposed form. Normal form C (NFC) first applies a canonical decomposition, then composes pre-combined characters again. In addition to these two forms, there are two additional normal forms based on compatibility equivalence. In Unicode, certain characters are supported which normally would be unified with other characters. For example, U+2160 (ROMAN NUMERAL ONE) is really the same thing as U+0049 (LATIN CAPITAL LETTER I). However, it is supported in Unicode for compatibility with existing character sets (e.g. gb2312). The normal form KD (NFKD) will apply the compatibility decomposition, i.e. replace all compatibility characters with their equivalents. The normal form KC (NFKC) first applies the compatibility decomposition, followed by the canonical composition. Even if two unicode strings are normalized and look the same to a human reader, if one has combining characters and the other doesn't, they may not compare equal. In addition, the module exposes the following constant: .. data:: unidata_version The version of the Unicode database used in this module. .. data:: ucd_3_2_0 This is an object that has the same methods as the entire module, but uses the Unicode database version 3.2 instead, for applications that require this specific version of the Unicode database (such as IDNA). Examples: >>> import unicodedata >>> unicodedata.lookup('LEFT CURLY BRACKET') '{' >>> unicodedata.name('/') 'SOLIDUS' >>> unicodedata.decimal('9') 9 >>> unicodedata.decimal('a') Traceback (most recent call last): File "", line 1, in ValueError: not a decimal >>> unicodedata.category('A') # 'L'etter, 'u'ppercase 'Lu' >>> unicodedata.bidirectional('\u0660') # 'A'rabic, 'N'umber 'AN' .. rubric:: Footnotes .. [#] http://www.unicode.org/Public/9.0.0/ucd/NameAliases.txt .. [#] http://www.unicode.org/Public/9.0.0/ucd/NamedSequences.txt PK 3]S  library/poplib.rst.txtnu[:mod:`poplib` --- POP3 protocol client ====================================== .. module:: poplib :synopsis: POP3 protocol client (requires sockets). .. sectionauthor:: Andrew T. Csillag .. revised by ESR, January 2000 **Source code:** :source:`Lib/poplib.py` .. index:: pair: POP3; protocol -------------- This module defines a class, :class:`POP3`, which encapsulates a connection to a POP3 server and implements the protocol as defined in :rfc:`1939`. The :class:`POP3` class supports both the minimal and optional command sets from :rfc:`1939`. The :class:`POP3` class also supports the ``STLS`` command introduced in :rfc:`2595` to enable encrypted communication on an already established connection. Additionally, this module provides a class :class:`POP3_SSL`, which provides support for connecting to POP3 servers that use SSL as an underlying protocol layer. Note that POP3, though widely supported, is obsolescent. The implementation quality of POP3 servers varies widely, and too many are quite poor. If your mailserver supports IMAP, you would be better off using the :class:`imaplib.IMAP4` class, as IMAP servers tend to be better implemented. The :mod:`poplib` module provides two classes: .. class:: POP3(host, port=POP3_PORT[, timeout]) This class implements the actual POP3 protocol. The connection is created when the instance is initialized. If *port* is omitted, the standard POP3 port (110) is used. The optional *timeout* parameter specifies a timeout in seconds for the connection attempt (if not specified, the global default timeout setting will be used). .. class:: POP3_SSL(host, port=POP3_SSL_PORT, keyfile=None, certfile=None, timeout=None, context=None) This is a subclass of :class:`POP3` that connects to the server over an SSL encrypted socket. If *port* is not specified, 995, the standard POP3-over-SSL port is used. *timeout* works as in the :class:`POP3` constructor. *context* is an optional :class:`ssl.SSLContext` object which allows bundling SSL configuration options, certificates and private keys into a single (potentially long-lived) structure. Please read :ref:`ssl-security` for best practices. *keyfile* and *certfile* are a legacy alternative to *context* - they can point to PEM-formatted private key and certificate chain files, respectively, for the SSL connection. .. versionchanged:: 3.2 *context* parameter added. .. versionchanged:: 3.4 The class now supports hostname check with :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see :data:`ssl.HAS_SNI`). .. deprecated:: 3.6 *keyfile* and *certfile* are deprecated in favor of *context*. Please use :meth:`ssl.SSLContext.load_cert_chain` instead, or let :func:`ssl.create_default_context` select the system's trusted CA certificates for you. One exception is defined as an attribute of the :mod:`poplib` module: .. exception:: error_proto Exception raised on any errors from this module (errors from :mod:`socket` module are not caught). The reason for the exception is passed to the constructor as a string. .. seealso:: Module :mod:`imaplib` The standard Python IMAP module. `Frequently Asked Questions About Fetchmail `_ The FAQ for the :program:`fetchmail` POP/IMAP client collects information on POP3 server variations and RFC noncompliance that may be useful if you need to write an application based on the POP protocol. .. _pop3-objects: POP3 Objects ------------ All POP3 commands are represented by methods of the same name, in lower-case; most return the response text sent by the server. An :class:`POP3` instance has the following methods: .. method:: POP3.set_debuglevel(level) Set the instance's debugging level. This controls the amount of debugging output printed. The default, ``0``, produces no debugging output. A value of ``1`` produces a moderate amount of debugging output, generally a single line per request. A value of ``2`` or higher produces the maximum amount of debugging output, logging each line sent and received on the control connection. .. method:: POP3.getwelcome() Returns the greeting string sent by the POP3 server. .. method:: POP3.capa() Query the server's capabilities as specified in :rfc:`2449`. Returns a dictionary in the form ``{'name': ['param'...]}``. .. versionadded:: 3.4 .. method:: POP3.user(username) Send user command, response should indicate that a password is required. .. method:: POP3.pass_(password) Send password, response includes message count and mailbox size. Note: the mailbox on the server is locked until :meth:`~poplib.quit` is called. .. method:: POP3.apop(user, secret) Use the more secure APOP authentication to log into the POP3 server. .. method:: POP3.rpop(user) Use RPOP authentication (similar to UNIX r-commands) to log into POP3 server. .. method:: POP3.stat() Get mailbox status. The result is a tuple of 2 integers: ``(message count, mailbox size)``. .. method:: POP3.list([which]) Request message list, result is in the form ``(response, ['mesg_num octets', ...], octets)``. If *which* is set, it is the message to list. .. method:: POP3.retr(which) Retrieve whole message number *which*, and set its seen flag. Result is in form ``(response, ['line', ...], octets)``. .. method:: POP3.dele(which) Flag message number *which* for deletion. On most servers deletions are not actually performed until QUIT (the major exception is Eudora QPOP, which deliberately violates the RFCs by doing pending deletes on any disconnect). .. method:: POP3.rset() Remove any deletion marks for the mailbox. .. method:: POP3.noop() Do nothing. Might be used as a keep-alive. .. method:: POP3.quit() Signoff: commit changes, unlock mailbox, drop connection. .. method:: POP3.top(which, howmuch) Retrieves the message header plus *howmuch* lines of the message after the header of message number *which*. Result is in form ``(response, ['line', ...], octets)``. The POP3 TOP command this method uses, unlike the RETR command, doesn't set the message's seen flag; unfortunately, TOP is poorly specified in the RFCs and is frequently broken in off-brand servers. Test this method by hand against the POP3 servers you will use before trusting it. .. method:: POP3.uidl(which=None) Return message digest (unique id) list. If *which* is specified, result contains the unique id for that message in the form ``'response mesgnum uid``, otherwise result is list ``(response, ['mesgnum uid', ...], octets)``. .. method:: POP3.utf8() Try to switch to UTF-8 mode. Returns the server response if successful, raises :class:`error_proto` if not. Specified in :RFC:`6856`. .. versionadded:: 3.5 .. method:: POP3.stls(context=None) Start a TLS session on the active connection as specified in :rfc:`2595`. This is only allowed before user authentication *context* parameter is a :class:`ssl.SSLContext` object which allows bundling SSL configuration options, certificates and private keys into a single (potentially long-lived) structure. Please read :ref:`ssl-security` for best practices. This method supports hostname checking via :attr:`ssl.SSLContext.check_hostname` and *Server Name Indication* (see :data:`ssl.HAS_SNI`). .. versionadded:: 3.4 Instances of :class:`POP3_SSL` have no additional methods. The interface of this subclass is identical to its parent. .. _pop3-example: POP3 Example ------------ Here is a minimal example (without error checking) that opens a mailbox and retrieves and prints all messages:: import getpass, poplib M = poplib.POP3('localhost') M.user(getpass.getuser()) M.pass_(getpass.getpass()) numMessages = len(M.list()[1]) for i in range(numMessages): for j in M.retr(i+1)[1]: print(j) At the end of the module, there is a test section that contains a more extensive example of usage. PK 3]:library/winsound.rst.txtnu[:mod:`winsound` --- Sound-playing interface for Windows ======================================================= .. module:: winsound :platform: Windows :synopsis: Access to the sound-playing machinery for Windows. .. moduleauthor:: Toby Dickenson .. sectionauthor:: Fred L. Drake, Jr. -------------- The :mod:`winsound` module provides access to the basic sound-playing machinery provided by Windows platforms. It includes functions and several constants. .. function:: Beep(frequency, duration) Beep the PC's speaker. The *frequency* parameter specifies frequency, in hertz, of the sound, and must be in the range 37 through 32,767. The *duration* parameter specifies the number of milliseconds the sound should last. If the system is not able to beep the speaker, :exc:`RuntimeError` is raised. .. function:: PlaySound(sound, flags) Call the underlying :c:func:`PlaySound` function from the Platform API. The *sound* parameter may be a filename, a system sound alias, audio data as a :term:`bytes-like object`, or ``None``. Its interpretation depends on the value of *flags*, which can be a bitwise ORed combination of the constants described below. If the *sound* parameter is ``None``, any currently playing waveform sound is stopped. If the system indicates an error, :exc:`RuntimeError` is raised. .. function:: MessageBeep(type=MB_OK) Call the underlying :c:func:`MessageBeep` function from the Platform API. This plays a sound as specified in the registry. The *type* argument specifies which sound to play; possible values are ``-1``, ``MB_ICONASTERISK``, ``MB_ICONEXCLAMATION``, ``MB_ICONHAND``, ``MB_ICONQUESTION``, and ``MB_OK``, all described below. The value ``-1`` produces a "simple beep"; this is the final fallback if a sound cannot be played otherwise. If the system indicates an error, :exc:`RuntimeError` is raised. .. data:: SND_FILENAME The *sound* parameter is the name of a WAV file. Do not use with :const:`SND_ALIAS`. .. data:: SND_ALIAS The *sound* parameter is a sound association name from the registry. If the registry contains no such name, play the system default sound unless :const:`SND_NODEFAULT` is also specified. If no default sound is registered, raise :exc:`RuntimeError`. Do not use with :const:`SND_FILENAME`. All Win32 systems support at least the following; most systems support many more: +--------------------------+----------------------------------------+ | :func:`PlaySound` *name* | Corresponding Control Panel Sound name | +==========================+========================================+ | ``'SystemAsterisk'`` | Asterisk | +--------------------------+----------------------------------------+ | ``'SystemExclamation'`` | Exclamation | +--------------------------+----------------------------------------+ | ``'SystemExit'`` | Exit Windows | +--------------------------+----------------------------------------+ | ``'SystemHand'`` | Critical Stop | +--------------------------+----------------------------------------+ | ``'SystemQuestion'`` | Question | +--------------------------+----------------------------------------+ For example:: import winsound # Play Windows exit sound. winsound.PlaySound("SystemExit", winsound.SND_ALIAS) # Probably play Windows default sound, if any is registered (because # "*" probably isn't the registered name of any sound). winsound.PlaySound("*", winsound.SND_ALIAS) .. data:: SND_LOOP Play the sound repeatedly. The :const:`SND_ASYNC` flag must also be used to avoid blocking. Cannot be used with :const:`SND_MEMORY`. .. data:: SND_MEMORY The *sound* parameter to :func:`PlaySound` is a memory image of a WAV file, as a :term:`bytes-like object`. .. note:: This module does not support playing from a memory image asynchronously, so a combination of this flag and :const:`SND_ASYNC` will raise :exc:`RuntimeError`. .. data:: SND_PURGE Stop playing all instances of the specified sound. .. note:: This flag is not supported on modern Windows platforms. .. data:: SND_ASYNC Return immediately, allowing sounds to play asynchronously. .. data:: SND_NODEFAULT If the specified sound cannot be found, do not play the system default sound. .. data:: SND_NOSTOP Do not interrupt sounds currently playing. .. data:: SND_NOWAIT Return immediately if the sound driver is busy. .. note:: This flag is not supported on modern Windows platforms. .. data:: MB_ICONASTERISK Play the ``SystemDefault`` sound. .. data:: MB_ICONEXCLAMATION Play the ``SystemExclamation`` sound. .. data:: MB_ICONHAND Play the ``SystemHand`` sound. .. data:: MB_ICONQUESTION Play the ``SystemQuestion`` sound. .. data:: MB_OK Play the ``SystemDefault`` sound. PK 3]@library/io.rst.txtnu[:mod:`io` --- Core tools for working with streams ================================================= .. module:: io :synopsis: Core tools for working with streams. .. moduleauthor:: Guido van Rossum .. moduleauthor:: Mike Verdone .. moduleauthor:: Mark Russell .. moduleauthor:: Antoine Pitrou .. moduleauthor:: Amaury Forgeot d'Arc .. moduleauthor:: Benjamin Peterson .. sectionauthor:: Benjamin Peterson **Source code:** :source:`Lib/io.py` -------------- .. _io-overview: Overview -------- .. index:: single: file object; io module The :mod:`io` module provides Python's main facilities for dealing with various types of I/O. There are three main types of I/O: *text I/O*, *binary I/O* and *raw I/O*. These are generic categories, and various backing stores can be used for each of them. A concrete object belonging to any of these categories is called a :term:`file object`. Other common terms are *stream* and *file-like object*. Independently of its category, each concrete stream object will also have various capabilities: it can be read-only, write-only, or read-write. It can also allow arbitrary random access (seeking forwards or backwards to any location), or only sequential access (for example in the case of a socket or pipe). All streams are careful about the type of data you give to them. For example giving a :class:`str` object to the ``write()`` method of a binary stream will raise a ``TypeError``. So will giving a :class:`bytes` object to the ``write()`` method of a text stream. .. versionchanged:: 3.3 Operations that used to raise :exc:`IOError` now raise :exc:`OSError`, since :exc:`IOError` is now an alias of :exc:`OSError`. Text I/O ^^^^^^^^ Text I/O expects and produces :class:`str` objects. This means that whenever the backing store is natively made of bytes (such as in the case of a file), encoding and decoding of data is made transparently as well as optional translation of platform-specific newline characters. The easiest way to create a text stream is with :meth:`open()`, optionally specifying an encoding:: f = open("myfile.txt", "r", encoding="utf-8") In-memory text streams are also available as :class:`StringIO` objects:: f = io.StringIO("some initial text data") The text stream API is described in detail in the documentation of :class:`TextIOBase`. Binary I/O ^^^^^^^^^^ Binary I/O (also called *buffered I/O*) expects :term:`bytes-like objects ` and produces :class:`bytes` objects. No encoding, decoding, or newline translation is performed. This category of streams can be used for all kinds of non-text data, and also when manual control over the handling of text data is desired. The easiest way to create a binary stream is with :meth:`open()` with ``'b'`` in the mode string:: f = open("myfile.jpg", "rb") In-memory binary streams are also available as :class:`BytesIO` objects:: f = io.BytesIO(b"some initial binary data: \x00\x01") The binary stream API is described in detail in the docs of :class:`BufferedIOBase`. Other library modules may provide additional ways to create text or binary streams. See :meth:`socket.socket.makefile` for example. Raw I/O ^^^^^^^ Raw I/O (also called *unbuffered I/O*) is generally used as a low-level building-block for binary and text streams; it is rarely useful to directly manipulate a raw stream from user code. Nevertheless, you can create a raw stream by opening a file in binary mode with buffering disabled:: f = open("myfile.jpg", "rb", buffering=0) The raw stream API is described in detail in the docs of :class:`RawIOBase`. High-level Module Interface --------------------------- .. data:: DEFAULT_BUFFER_SIZE An int containing the default buffer size used by the module's buffered I/O classes. :func:`open` uses the file's blksize (as obtained by :func:`os.stat`) if possible. .. function:: open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None) This is an alias for the builtin :func:`open` function. .. exception:: BlockingIOError This is a compatibility alias for the builtin :exc:`BlockingIOError` exception. .. exception:: UnsupportedOperation An exception inheriting :exc:`OSError` and :exc:`ValueError` that is raised when an unsupported operation is called on a stream. In-memory streams ^^^^^^^^^^^^^^^^^ It is also possible to use a :class:`str` or :term:`bytes-like object` as a file for both reading and writing. For strings :class:`StringIO` can be used like a file opened in text mode. :class:`BytesIO` can be used like a file opened in binary mode. Both provide full read-write capabilities with random access. .. seealso:: :mod:`sys` contains the standard IO streams: :data:`sys.stdin`, :data:`sys.stdout`, and :data:`sys.stderr`. Class hierarchy --------------- The implementation of I/O streams is organized as a hierarchy of classes. First :term:`abstract base classes ` (ABCs), which are used to specify the various categories of streams, then concrete classes providing the standard stream implementations. .. note:: The abstract base classes also provide default implementations of some methods in order to help implementation of concrete stream classes. For example, :class:`BufferedIOBase` provides unoptimized implementations of :meth:`~IOBase.readinto` and :meth:`~IOBase.readline`. At the top of the I/O hierarchy is the abstract base class :class:`IOBase`. It defines the basic interface to a stream. Note, however, that there is no separation between reading and writing to streams; implementations are allowed to raise :exc:`UnsupportedOperation` if they do not support a given operation. The :class:`RawIOBase` ABC extends :class:`IOBase`. It deals with the reading and writing of bytes to a stream. :class:`FileIO` subclasses :class:`RawIOBase` to provide an interface to files in the machine's file system. The :class:`BufferedIOBase` ABC deals with buffering on a raw byte stream (:class:`RawIOBase`). Its subclasses, :class:`BufferedWriter`, :class:`BufferedReader`, and :class:`BufferedRWPair` buffer streams that are readable, writable, and both readable and writable. :class:`BufferedRandom` provides a buffered interface to random access streams. Another :class:`BufferedIOBase` subclass, :class:`BytesIO`, is a stream of in-memory bytes. The :class:`TextIOBase` ABC, another subclass of :class:`IOBase`, deals with streams whose bytes represent text, and handles encoding and decoding to and from strings. :class:`TextIOWrapper`, which extends it, is a buffered text interface to a buffered raw stream (:class:`BufferedIOBase`). Finally, :class:`StringIO` is an in-memory stream for text. Argument names are not part of the specification, and only the arguments of :func:`open` are intended to be used as keyword arguments. The following table summarizes the ABCs provided by the :mod:`io` module: .. tabularcolumns:: |l|l|L|L| ========================= ================== ======================== ================================================== ABC Inherits Stub Methods Mixin Methods and Properties ========================= ================== ======================== ================================================== :class:`IOBase` ``fileno``, ``seek``, ``close``, ``closed``, ``__enter__``, and ``truncate`` ``__exit__``, ``flush``, ``isatty``, ``__iter__``, ``__next__``, ``readable``, ``readline``, ``readlines``, ``seekable``, ``tell``, ``writable``, and ``writelines`` :class:`RawIOBase` :class:`IOBase` ``readinto`` and Inherited :class:`IOBase` methods, ``read``, ``write`` and ``readall`` :class:`BufferedIOBase` :class:`IOBase` ``detach``, ``read``, Inherited :class:`IOBase` methods, ``readinto``, ``read1``, and ``write`` and ``readinto1`` :class:`TextIOBase` :class:`IOBase` ``detach``, ``read``, Inherited :class:`IOBase` methods, ``encoding``, ``readline``, and ``errors``, and ``newlines`` ``write`` ========================= ================== ======================== ================================================== I/O Base Classes ^^^^^^^^^^^^^^^^ .. class:: IOBase The abstract base class for all I/O classes, acting on streams of bytes. There is no public constructor. This class provides empty abstract implementations for many methods that derived classes can override selectively; the default implementations represent a file that cannot be read, written or seeked. Even though :class:`IOBase` does not declare :meth:`read`, :meth:`readinto`, or :meth:`write` because their signatures will vary, implementations and clients should consider those methods part of the interface. Also, implementations may raise a :exc:`ValueError` (or :exc:`UnsupportedOperation`) when operations they do not support are called. The basic type used for binary data read from or written to a file is :class:`bytes`. Other :term:`bytes-like objects ` are accepted as method arguments too. In some cases, such as :meth:`~RawIOBase.readinto`, a writable object such as :class:`bytearray` is required. Text I/O classes work with :class:`str` data. Note that calling any method (even inquiries) on a closed stream is undefined. Implementations may raise :exc:`ValueError` in this case. :class:`IOBase` (and its subclasses) supports the iterator protocol, meaning that an :class:`IOBase` object can be iterated over yielding the lines in a stream. Lines are defined slightly differently depending on whether the stream is a binary stream (yielding bytes), or a text stream (yielding character strings). See :meth:`~IOBase.readline` below. :class:`IOBase` is also a context manager and therefore supports the :keyword:`with` statement. In this example, *file* is closed after the :keyword:`with` statement's suite is finished---even if an exception occurs:: with open('spam.txt', 'w') as file: file.write('Spam and eggs!') :class:`IOBase` provides these data attributes and methods: .. method:: close() Flush and close this stream. This method has no effect if the file is already closed. Once the file is closed, any operation on the file (e.g. reading or writing) will raise a :exc:`ValueError`. As a convenience, it is allowed to call this method more than once; only the first call, however, will have an effect. .. attribute:: closed ``True`` if the stream is closed. .. method:: fileno() Return the underlying file descriptor (an integer) of the stream if it exists. An :exc:`OSError` is raised if the IO object does not use a file descriptor. .. method:: flush() Flush the write buffers of the stream if applicable. This does nothing for read-only and non-blocking streams. .. method:: isatty() Return ``True`` if the stream is interactive (i.e., connected to a terminal/tty device). .. method:: readable() Return ``True`` if the stream can be read from. If ``False``, :meth:`read` will raise :exc:`OSError`. .. method:: readline(size=-1) Read and return one line from the stream. If *size* is specified, at most *size* bytes will be read. The line terminator is always ``b'\n'`` for binary files; for text files, the *newline* argument to :func:`open` can be used to select the line terminator(s) recognized. .. method:: readlines(hint=-1) Read and return a list of lines from the stream. *hint* can be specified to control the number of lines read: no more lines will be read if the total size (in bytes/characters) of all lines so far exceeds *hint*. Note that it's already possible to iterate on file objects using ``for line in file: ...`` without calling ``file.readlines()``. .. method:: seek(offset[, whence]) Change the stream position to the given byte *offset*. *offset* is interpreted relative to the position indicated by *whence*. The default value for *whence* is :data:`SEEK_SET`. Values for *whence* are: * :data:`SEEK_SET` or ``0`` -- start of the stream (the default); *offset* should be zero or positive * :data:`SEEK_CUR` or ``1`` -- current stream position; *offset* may be negative * :data:`SEEK_END` or ``2`` -- end of the stream; *offset* is usually negative Return the new absolute position. .. versionadded:: 3.1 The ``SEEK_*`` constants. .. versionadded:: 3.3 Some operating systems could support additional values, like :data:`os.SEEK_HOLE` or :data:`os.SEEK_DATA`. The valid values for a file could depend on it being open in text or binary mode. .. method:: seekable() Return ``True`` if the stream supports random access. If ``False``, :meth:`seek`, :meth:`tell` and :meth:`truncate` will raise :exc:`OSError`. .. method:: tell() Return the current stream position. .. method:: truncate(size=None) Resize the stream to the given *size* in bytes (or the current position if *size* is not specified). The current stream position isn't changed. This resizing can extend or reduce the current file size. In case of extension, the contents of the new file area depend on the platform (on most systems, additional bytes are zero-filled). The new file size is returned. .. versionchanged:: 3.5 Windows will now zero-fill files when extending. .. method:: writable() Return ``True`` if the stream supports writing. If ``False``, :meth:`write` and :meth:`truncate` will raise :exc:`OSError`. .. method:: writelines(lines) Write a list of lines to the stream. Line separators are not added, so it is usual for each of the lines provided to have a line separator at the end. .. method:: __del__() Prepare for object destruction. :class:`IOBase` provides a default implementation of this method that calls the instance's :meth:`~IOBase.close` method. .. class:: RawIOBase Base class for raw binary I/O. It inherits :class:`IOBase`. There is no public constructor. Raw binary I/O typically provides low-level access to an underlying OS device or API, and does not try to encapsulate it in high-level primitives (this is left to Buffered I/O and Text I/O, described later in this page). In addition to the attributes and methods from :class:`IOBase`, :class:`RawIOBase` provides the following methods: .. method:: read(size=-1) Read up to *size* bytes from the object and return them. As a convenience, if *size* is unspecified or -1, all bytes until EOF are returned. Otherwise, only one system call is ever made. Fewer than *size* bytes may be returned if the operating system call returns fewer than *size* bytes. If 0 bytes are returned, and *size* was not 0, this indicates end of file. If the object is in non-blocking mode and no bytes are available, ``None`` is returned. The default implementation defers to :meth:`readall` and :meth:`readinto`. .. method:: readall() Read and return all the bytes from the stream until EOF, using multiple calls to the stream if necessary. .. method:: readinto(b) Read bytes into a pre-allocated, writable :term:`bytes-like object` *b*, and return the number of bytes read. If the object is in non-blocking mode and no bytes are available, ``None`` is returned. .. method:: write(b) Write the given :term:`bytes-like object`, *b*, to the underlying raw stream, and return the number of bytes written. This can be less than the length of *b* in bytes, depending on specifics of the underlying raw stream, and especially if it is in non-blocking mode. ``None`` is returned if the raw stream is set not to block and no single byte could be readily written to it. The caller may release or mutate *b* after this method returns, so the implementation should only access *b* during the method call. .. class:: BufferedIOBase Base class for binary streams that support some kind of buffering. It inherits :class:`IOBase`. There is no public constructor. The main difference with :class:`RawIOBase` is that methods :meth:`read`, :meth:`readinto` and :meth:`write` will try (respectively) to read as much input as requested or to consume all given output, at the expense of making perhaps more than one system call. In addition, those methods can raise :exc:`BlockingIOError` if the underlying raw stream is in non-blocking mode and cannot take or give enough data; unlike their :class:`RawIOBase` counterparts, they will never return ``None``. Besides, the :meth:`read` method does not have a default implementation that defers to :meth:`readinto`. A typical :class:`BufferedIOBase` implementation should not inherit from a :class:`RawIOBase` implementation, but wrap one, like :class:`BufferedWriter` and :class:`BufferedReader` do. :class:`BufferedIOBase` provides or overrides these methods and attribute in addition to those from :class:`IOBase`: .. attribute:: raw The underlying raw stream (a :class:`RawIOBase` instance) that :class:`BufferedIOBase` deals with. This is not part of the :class:`BufferedIOBase` API and may not exist on some implementations. .. method:: detach() Separate the underlying raw stream from the buffer and return it. After the raw stream has been detached, the buffer is in an unusable state. Some buffers, like :class:`BytesIO`, do not have the concept of a single raw stream to return from this method. They raise :exc:`UnsupportedOperation`. .. versionadded:: 3.1 .. method:: read(size=-1) Read and return up to *size* bytes. If the argument is omitted, ``None``, or negative, data is read and returned until EOF is reached. An empty :class:`bytes` object is returned if the stream is already at EOF. If the argument is positive, and the underlying raw stream is not interactive, multiple raw reads may be issued to satisfy the byte count (unless EOF is reached first). But for interactive raw streams, at most one raw read will be issued, and a short result does not imply that EOF is imminent. A :exc:`BlockingIOError` is raised if the underlying raw stream is in non blocking-mode, and has no data available at the moment. .. method:: read1(size=-1) Read and return up to *size* bytes, with at most one call to the underlying raw stream's :meth:`~RawIOBase.read` (or :meth:`~RawIOBase.readinto`) method. This can be useful if you are implementing your own buffering on top of a :class:`BufferedIOBase` object. .. method:: readinto(b) Read bytes into a pre-allocated, writable :term:`bytes-like object` *b* and return the number of bytes read. Like :meth:`read`, multiple reads may be issued to the underlying raw stream, unless the latter is interactive. A :exc:`BlockingIOError` is raised if the underlying raw stream is in non blocking-mode, and has no data available at the moment. .. method:: readinto1(b) Read bytes into a pre-allocated, writable :term:`bytes-like object` *b*, using at most one call to the underlying raw stream's :meth:`~RawIOBase.read` (or :meth:`~RawIOBase.readinto`) method. Return the number of bytes read. A :exc:`BlockingIOError` is raised if the underlying raw stream is in non blocking-mode, and has no data available at the moment. .. versionadded:: 3.5 .. method:: write(b) Write the given :term:`bytes-like object`, *b*, and return the number of bytes written (always equal to the length of *b* in bytes, since if the write fails an :exc:`OSError` will be raised). Depending on the actual implementation, these bytes may be readily written to the underlying stream, or held in a buffer for performance and latency reasons. When in non-blocking mode, a :exc:`BlockingIOError` is raised if the data needed to be written to the raw stream but it couldn't accept all the data without blocking. The caller may release or mutate *b* after this method returns, so the implementation should only access *b* during the method call. Raw File I/O ^^^^^^^^^^^^ .. class:: FileIO(name, mode='r', closefd=True, opener=None) :class:`FileIO` represents an OS-level file containing bytes data. It implements the :class:`RawIOBase` interface (and therefore the :class:`IOBase` interface, too). The *name* can be one of two things: * a character string or :class:`bytes` object representing the path to the file which will be opened. In this case closefd must be ``True`` (the default) otherwise an error will be raised. * an integer representing the number of an existing OS-level file descriptor to which the resulting :class:`FileIO` object will give access. When the FileIO object is closed this fd will be closed as well, unless *closefd* is set to ``False``. The *mode* can be ``'r'``, ``'w'``, ``'x'`` or ``'a'`` for reading (default), writing, exclusive creation or appending. The file will be created if it doesn't exist when opened for writing or appending; it will be truncated when opened for writing. :exc:`FileExistsError` will be raised if it already exists when opened for creating. Opening a file for creating implies writing, so this mode behaves in a similar way to ``'w'``. Add a ``'+'`` to the mode to allow simultaneous reading and writing. The :meth:`read` (when called with a positive argument), :meth:`readinto` and :meth:`write` methods on this class will only make one system call. A custom opener can be used by passing a callable as *opener*. The underlying file descriptor for the file object is then obtained by calling *opener* with (*name*, *flags*). *opener* must return an open file descriptor (passing :mod:`os.open` as *opener* results in functionality similar to passing ``None``). The newly created file is :ref:`non-inheritable `. See the :func:`open` built-in function for examples on using the *opener* parameter. .. versionchanged:: 3.3 The *opener* parameter was added. The ``'x'`` mode was added. .. versionchanged:: 3.4 The file is now non-inheritable. In addition to the attributes and methods from :class:`IOBase` and :class:`RawIOBase`, :class:`FileIO` provides the following data attributes: .. attribute:: mode The mode as given in the constructor. .. attribute:: name The file name. This is the file descriptor of the file when no name is given in the constructor. Buffered Streams ^^^^^^^^^^^^^^^^ Buffered I/O streams provide a higher-level interface to an I/O device than raw I/O does. .. class:: BytesIO([initial_bytes]) A stream implementation using an in-memory bytes buffer. It inherits :class:`BufferedIOBase`. The buffer is discarded when the :meth:`~IOBase.close` method is called. The optional argument *initial_bytes* is a :term:`bytes-like object` that contains initial data. :class:`BytesIO` provides or overrides these methods in addition to those from :class:`BufferedIOBase` and :class:`IOBase`: .. method:: getbuffer() Return a readable and writable view over the contents of the buffer without copying them. Also, mutating the view will transparently update the contents of the buffer:: >>> b = io.BytesIO(b"abcdef") >>> view = b.getbuffer() >>> view[2:4] = b"56" >>> b.getvalue() b'ab56ef' .. note:: As long as the view exists, the :class:`BytesIO` object cannot be resized or closed. .. versionadded:: 3.2 .. method:: getvalue() Return :class:`bytes` containing the entire contents of the buffer. .. method:: read1() In :class:`BytesIO`, this is the same as :meth:`read`. .. method:: readinto1() In :class:`BytesIO`, this is the same as :meth:`readinto`. .. versionadded:: 3.5 .. class:: BufferedReader(raw, buffer_size=DEFAULT_BUFFER_SIZE) A buffer providing higher-level access to a readable, sequential :class:`RawIOBase` object. It inherits :class:`BufferedIOBase`. When reading data from this object, a larger amount of data may be requested from the underlying raw stream, and kept in an internal buffer. The buffered data can then be returned directly on subsequent reads. The constructor creates a :class:`BufferedReader` for the given readable *raw* stream and *buffer_size*. If *buffer_size* is omitted, :data:`DEFAULT_BUFFER_SIZE` is used. :class:`BufferedReader` provides or overrides these methods in addition to those from :class:`BufferedIOBase` and :class:`IOBase`: .. method:: peek([size]) Return bytes from the stream without advancing the position. At most one single read on the raw stream is done to satisfy the call. The number of bytes returned may be less or more than requested. .. method:: read([size]) Read and return *size* bytes, or if *size* is not given or negative, until EOF or if the read call would block in non-blocking mode. .. method:: read1(size) Read and return up to *size* bytes with only one call on the raw stream. If at least one byte is buffered, only buffered bytes are returned. Otherwise, one raw stream read call is made. .. class:: BufferedWriter(raw, buffer_size=DEFAULT_BUFFER_SIZE) A buffer providing higher-level access to a writeable, sequential :class:`RawIOBase` object. It inherits :class:`BufferedIOBase`. When writing to this object, data is normally placed into an internal buffer. The buffer will be written out to the underlying :class:`RawIOBase` object under various conditions, including: * when the buffer gets too small for all pending data; * when :meth:`flush()` is called; * when a :meth:`seek()` is requested (for :class:`BufferedRandom` objects); * when the :class:`BufferedWriter` object is closed or destroyed. The constructor creates a :class:`BufferedWriter` for the given writeable *raw* stream. If the *buffer_size* is not given, it defaults to :data:`DEFAULT_BUFFER_SIZE`. :class:`BufferedWriter` provides or overrides these methods in addition to those from :class:`BufferedIOBase` and :class:`IOBase`: .. method:: flush() Force bytes held in the buffer into the raw stream. A :exc:`BlockingIOError` should be raised if the raw stream blocks. .. method:: write(b) Write the :term:`bytes-like object`, *b*, and return the number of bytes written. When in non-blocking mode, a :exc:`BlockingIOError` is raised if the buffer needs to be written out but the raw stream blocks. .. class:: BufferedRandom(raw, buffer_size=DEFAULT_BUFFER_SIZE) A buffered interface to random access streams. It inherits :class:`BufferedReader` and :class:`BufferedWriter`, and further supports :meth:`seek` and :meth:`tell` functionality. The constructor creates a reader and writer for a seekable raw stream, given in the first argument. If the *buffer_size* is omitted it defaults to :data:`DEFAULT_BUFFER_SIZE`. :class:`BufferedRandom` is capable of anything :class:`BufferedReader` or :class:`BufferedWriter` can do. .. class:: BufferedRWPair(reader, writer, buffer_size=DEFAULT_BUFFER_SIZE) A buffered I/O object combining two unidirectional :class:`RawIOBase` objects -- one readable, the other writeable -- into a single bidirectional endpoint. It inherits :class:`BufferedIOBase`. *reader* and *writer* are :class:`RawIOBase` objects that are readable and writeable respectively. If the *buffer_size* is omitted it defaults to :data:`DEFAULT_BUFFER_SIZE`. :class:`BufferedRWPair` implements all of :class:`BufferedIOBase`\'s methods except for :meth:`~BufferedIOBase.detach`, which raises :exc:`UnsupportedOperation`. .. warning:: :class:`BufferedRWPair` does not attempt to synchronize accesses to its underlying raw streams. You should not pass it the same object as reader and writer; use :class:`BufferedRandom` instead. Text I/O ^^^^^^^^ .. class:: TextIOBase Base class for text streams. This class provides a character and line based interface to stream I/O. There is no :meth:`readinto` method because Python's character strings are immutable. It inherits :class:`IOBase`. There is no public constructor. :class:`TextIOBase` provides or overrides these data attributes and methods in addition to those from :class:`IOBase`: .. attribute:: encoding The name of the encoding used to decode the stream's bytes into strings, and to encode strings into bytes. .. attribute:: errors The error setting of the decoder or encoder. .. attribute:: newlines A string, a tuple of strings, or ``None``, indicating the newlines translated so far. Depending on the implementation and the initial constructor flags, this may not be available. .. attribute:: buffer The underlying binary buffer (a :class:`BufferedIOBase` instance) that :class:`TextIOBase` deals with. This is not part of the :class:`TextIOBase` API and may not exist in some implementations. .. method:: detach() Separate the underlying binary buffer from the :class:`TextIOBase` and return it. After the underlying buffer has been detached, the :class:`TextIOBase` is in an unusable state. Some :class:`TextIOBase` implementations, like :class:`StringIO`, may not have the concept of an underlying buffer and calling this method will raise :exc:`UnsupportedOperation`. .. versionadded:: 3.1 .. method:: read(size=-1) Read and return at most *size* characters from the stream as a single :class:`str`. If *size* is negative or ``None``, reads until EOF. .. method:: readline(size=-1) Read until newline or EOF and return a single ``str``. If the stream is already at EOF, an empty string is returned. If *size* is specified, at most *size* characters will be read. .. method:: seek(offset[, whence]) Change the stream position to the given *offset*. Behaviour depends on the *whence* parameter. The default value for *whence* is :data:`SEEK_SET`. * :data:`SEEK_SET` or ``0``: seek from the start of the stream (the default); *offset* must either be a number returned by :meth:`TextIOBase.tell`, or zero. Any other *offset* value produces undefined behaviour. * :data:`SEEK_CUR` or ``1``: "seek" to the current position; *offset* must be zero, which is a no-operation (all other values are unsupported). * :data:`SEEK_END` or ``2``: seek to the end of the stream; *offset* must be zero (all other values are unsupported). Return the new absolute position as an opaque number. .. versionadded:: 3.1 The ``SEEK_*`` constants. .. method:: tell() Return the current stream position as an opaque number. The number does not usually represent a number of bytes in the underlying binary storage. .. method:: write(s) Write the string *s* to the stream and return the number of characters written. .. class:: TextIOWrapper(buffer, encoding=None, errors=None, newline=None, \ line_buffering=False, write_through=False) A buffered text stream over a :class:`BufferedIOBase` binary stream. It inherits :class:`TextIOBase`. *encoding* gives the name of the encoding that the stream will be decoded or encoded with. It defaults to :func:`locale.getpreferredencoding(False) `. *errors* is an optional string that specifies how encoding and decoding errors are to be handled. Pass ``'strict'`` to raise a :exc:`ValueError` exception if there is an encoding error (the default of ``None`` has the same effect), or pass ``'ignore'`` to ignore errors. (Note that ignoring encoding errors can lead to data loss.) ``'replace'`` causes a replacement marker (such as ``'?'``) to be inserted where there is malformed data. ``'backslashreplace'`` causes malformed data to be replaced by a backslashed escape sequence. When writing, ``'xmlcharrefreplace'`` (replace with the appropriate XML character reference) or ``'namereplace'`` (replace with ``\N{...}`` escape sequences) can be used. Any other error handling name that has been registered with :func:`codecs.register_error` is also valid. .. index:: single: universal newlines; io.TextIOWrapper class *newline* controls how line endings are handled. It can be ``None``, ``''``, ``'\n'``, ``'\r'``, and ``'\r\n'``. It works as follows: * When reading input from the stream, if *newline* is ``None``, :term:`universal newlines` mode is enabled. Lines in the input can end in ``'\n'``, ``'\r'``, or ``'\r\n'``, and these are translated into ``'\n'`` before being returned to the caller. If it is ``''``, universal newlines mode is enabled, but line endings are returned to the caller untranslated. If it has any of the other legal values, input lines are only terminated by the given string, and the line ending is returned to the caller untranslated. * When writing output to the stream, if *newline* is ``None``, any ``'\n'`` characters written are translated to the system default line separator, :data:`os.linesep`. If *newline* is ``''`` or ``'\n'``, no translation takes place. If *newline* is any of the other legal values, any ``'\n'`` characters written are translated to the given string. If *line_buffering* is ``True``, :meth:`flush` is implied when a call to write contains a newline character or a carriage return. If *write_through* is ``True``, calls to :meth:`write` are guaranteed not to be buffered: any data written on the :class:`TextIOWrapper` object is immediately handled to its underlying binary *buffer*. .. versionchanged:: 3.3 The *write_through* argument has been added. .. versionchanged:: 3.3 The default *encoding* is now ``locale.getpreferredencoding(False)`` instead of ``locale.getpreferredencoding()``. Don't change temporary the locale encoding using :func:`locale.setlocale`, use the current locale encoding instead of the user preferred encoding. :class:`TextIOWrapper` provides one attribute in addition to those of :class:`TextIOBase` and its parents: .. attribute:: line_buffering Whether line buffering is enabled. .. class:: StringIO(initial_value='', newline='\\n') An in-memory stream for text I/O. The text buffer is discarded when the :meth:`~IOBase.close` method is called. The initial value of the buffer can be set by providing *initial_value*. If newline translation is enabled, newlines will be encoded as if by :meth:`~TextIOBase.write`. The stream is positioned at the start of the buffer. The *newline* argument works like that of :class:`TextIOWrapper`. The default is to consider only ``\n`` characters as ends of lines and to do no newline translation. If *newline* is set to ``None``, newlines are written as ``\n`` on all platforms, but universal newline decoding is still performed when reading. :class:`StringIO` provides this method in addition to those from :class:`TextIOBase` and its parents: .. method:: getvalue() Return a ``str`` containing the entire contents of the buffer. Newlines are decoded as if by :meth:`~TextIOBase.read`, although the stream position is not changed. Example usage:: import io output = io.StringIO() output.write('First line.\n') print('Second line.', file=output) # Retrieve file contents -- this will be # 'First line.\nSecond line.\n' contents = output.getvalue() # Close object and discard memory buffer -- # .getvalue() will now raise an exception. output.close() .. index:: single: universal newlines; io.IncrementalNewlineDecoder class .. class:: IncrementalNewlineDecoder A helper codec that decodes newlines for :term:`universal newlines` mode. It inherits :class:`codecs.IncrementalDecoder`. Performance ----------- This section discusses the performance of the provided concrete I/O implementations. Binary I/O ^^^^^^^^^^ By reading and writing only large chunks of data even when the user asks for a single byte, buffered I/O hides any inefficiency in calling and executing the operating system's unbuffered I/O routines. The gain depends on the OS and the kind of I/O which is performed. For example, on some modern OSes such as Linux, unbuffered disk I/O can be as fast as buffered I/O. The bottom line, however, is that buffered I/O offers predictable performance regardless of the platform and the backing device. Therefore, it is almost always preferable to use buffered I/O rather than unbuffered I/O for binary data. Text I/O ^^^^^^^^ Text I/O over a binary storage (such as a file) is significantly slower than binary I/O over the same storage, because it requires conversions between unicode and binary data using a character codec. This can become noticeable handling huge amounts of text data like large log files. Also, :meth:`TextIOWrapper.tell` and :meth:`TextIOWrapper.seek` are both quite slow due to the reconstruction algorithm used. :class:`StringIO`, however, is a native in-memory unicode container and will exhibit similar speed to :class:`BytesIO`. Multi-threading ^^^^^^^^^^^^^^^ :class:`FileIO` objects are thread-safe to the extent that the operating system calls (such as ``read(2)`` under Unix) they wrap are thread-safe too. Binary buffered objects (instances of :class:`BufferedReader`, :class:`BufferedWriter`, :class:`BufferedRandom` and :class:`BufferedRWPair`) protect their internal structures using a lock; it is therefore safe to call them from multiple threads at once. :class:`TextIOWrapper` objects are not thread-safe. Reentrancy ^^^^^^^^^^ Binary buffered objects (instances of :class:`BufferedReader`, :class:`BufferedWriter`, :class:`BufferedRandom` and :class:`BufferedRWPair`) are not reentrant. While reentrant calls will not happen in normal situations, they can arise from doing I/O in a :mod:`signal` handler. If a thread tries to re-enter a buffered object which it is already accessing, a :exc:`RuntimeError` is raised. Note this doesn't prohibit a different thread from entering the buffered object. The above implicitly extends to text files, since the :func:`open()` function will wrap a buffered object inside a :class:`TextIOWrapper`. This includes standard streams and therefore affects the built-in function :func:`print()` as well. PK 3]!clibrary/urllib.request.rst.txtnu[:mod:`urllib.request` --- Extensible library for opening URLs ============================================================= .. module:: urllib.request :synopsis: Extensible library for opening URLs. .. moduleauthor:: Jeremy Hylton .. sectionauthor:: Moshe Zadka .. sectionauthor:: Senthil Kumaran **Source code:** :source:`Lib/urllib/request.py` -------------- The :mod:`urllib.request` module defines functions and classes which help in opening URLs (mostly HTTP) in a complex world --- basic and digest authentication, redirections, cookies and more. .. seealso:: The `Requests package `_ is recommended for a higher-level HTTP client interface. The :mod:`urllib.request` module defines the following functions: .. function:: urlopen(url, data=None[, timeout], *, cafile=None, capath=None, cadefault=False, context=None) Open the URL *url*, which can be either a string or a :class:`Request` object. *data* must be an object specifying additional data to be sent to the server, or ``None`` if no such data is needed. See :class:`Request` for details. urllib.request module uses HTTP/1.1 and includes ``Connection:close`` header in its HTTP requests. The optional *timeout* parameter specifies a timeout in seconds for blocking operations like the connection attempt (if not specified, the global default timeout setting will be used). This actually only works for HTTP, HTTPS and FTP connections. If *context* is specified, it must be a :class:`ssl.SSLContext` instance describing the various SSL options. See :class:`~http.client.HTTPSConnection` for more details. The optional *cafile* and *capath* parameters specify a set of trusted CA certificates for HTTPS requests. *cafile* should point to a single file containing a bundle of CA certificates, whereas *capath* should point to a directory of hashed certificate files. More information can be found in :meth:`ssl.SSLContext.load_verify_locations`. The *cadefault* parameter is ignored. This function always returns an object which can work as a :term:`context manager` and has methods such as * :meth:`~urllib.response.addinfourl.geturl` --- return the URL of the resource retrieved, commonly used to determine if a redirect was followed * :meth:`~urllib.response.addinfourl.info` --- return the meta-information of the page, such as headers, in the form of an :func:`email.message_from_string` instance (see `Quick Reference to HTTP Headers `_) * :meth:`~urllib.response.addinfourl.getcode` -- return the HTTP status code of the response. For HTTP and HTTPS URLs, this function returns a :class:`http.client.HTTPResponse` object slightly modified. In addition to the three new methods above, the msg attribute contains the same information as the :attr:`~http.client.HTTPResponse.reason` attribute --- the reason phrase returned by server --- instead of the response headers as it is specified in the documentation for :class:`~http.client.HTTPResponse`. For FTP, file, and data URLs and requests explicitly handled by legacy :class:`URLopener` and :class:`FancyURLopener` classes, this function returns a :class:`urllib.response.addinfourl` object. Raises :exc:`~urllib.error.URLError` on protocol errors. Note that ``None`` may be returned if no handler handles the request (though the default installed global :class:`OpenerDirector` uses :class:`UnknownHandler` to ensure this never happens). In addition, if proxy settings are detected (for example, when a ``*_proxy`` environment variable like :envvar:`http_proxy` is set), :class:`ProxyHandler` is default installed and makes sure the requests are handled through the proxy. The legacy ``urllib.urlopen`` function from Python 2.6 and earlier has been discontinued; :func:`urllib.request.urlopen` corresponds to the old ``urllib2.urlopen``. Proxy handling, which was done by passing a dictionary parameter to ``urllib.urlopen``, can be obtained by using :class:`ProxyHandler` objects. .. versionchanged:: 3.2 *cafile* and *capath* were added. .. versionchanged:: 3.2 HTTPS virtual hosts are now supported if possible (that is, if :data:`ssl.HAS_SNI` is true). .. versionadded:: 3.2 *data* can be an iterable object. .. versionchanged:: 3.3 *cadefault* was added. .. versionchanged:: 3.4.3 *context* was added. .. deprecated:: 3.6 *cafile*, *capath* and *cadefault* are deprecated in favor of *context*. Please use :meth:`ssl.SSLContext.load_cert_chain` instead, or let :func:`ssl.create_default_context` select the system's trusted CA certificates for you. .. function:: install_opener(opener) Install an :class:`OpenerDirector` instance as the default global opener. Installing an opener is only necessary if you want urlopen to use that opener; otherwise, simply call :meth:`OpenerDirector.open` instead of :func:`~urllib.request.urlopen`. The code does not check for a real :class:`OpenerDirector`, and any class with the appropriate interface will work. .. function:: build_opener([handler, ...]) Return an :class:`OpenerDirector` instance, which chains the handlers in the order given. *handler*\s can be either instances of :class:`BaseHandler`, or subclasses of :class:`BaseHandler` (in which case it must be possible to call the constructor without any parameters). Instances of the following classes will be in front of the *handler*\s, unless the *handler*\s contain them, instances of them or subclasses of them: :class:`ProxyHandler` (if proxy settings are detected), :class:`UnknownHandler`, :class:`HTTPHandler`, :class:`HTTPDefaultErrorHandler`, :class:`HTTPRedirectHandler`, :class:`FTPHandler`, :class:`FileHandler`, :class:`HTTPErrorProcessor`. If the Python installation has SSL support (i.e., if the :mod:`ssl` module can be imported), :class:`HTTPSHandler` will also be added. A :class:`BaseHandler` subclass may also change its :attr:`handler_order` attribute to modify its position in the handlers list. .. function:: pathname2url(path) Convert the pathname *path* from the local syntax for a path to the form used in the path component of a URL. This does not produce a complete URL. The return value will already be quoted using the :func:`~urllib.parse.quote` function. .. function:: url2pathname(path) Convert the path component *path* from a percent-encoded URL to the local syntax for a path. This does not accept a complete URL. This function uses :func:`~urllib.parse.unquote` to decode *path*. .. function:: getproxies() This helper function returns a dictionary of scheme to proxy server URL mappings. It scans the environment for variables named ``_proxy``, in a case insensitive approach, for all operating systems first, and when it cannot find it, looks for proxy information from Mac OSX System Configuration for Mac OS X and Windows Systems Registry for Windows. If both lowercase and uppercase environment variables exist (and disagree), lowercase is preferred. .. note:: If the environment variable ``REQUEST_METHOD`` is set, which usually indicates your script is running in a CGI environment, the environment variable ``HTTP_PROXY`` (uppercase ``_PROXY``) will be ignored. This is because that variable can be injected by a client using the "Proxy:" HTTP header. If you need to use an HTTP proxy in a CGI environment, either use ``ProxyHandler`` explicitly, or make sure the variable name is in lowercase (or at least the ``_proxy`` suffix). The following classes are provided: .. class:: Request(url, data=None, headers={}, origin_req_host=None, unverifiable=False, method=None) This class is an abstraction of a URL request. *url* should be a string containing a valid URL. *data* must be an object specifying additional data to send to the server, or ``None`` if no such data is needed. Currently HTTP requests are the only ones that use *data*. The supported object types include bytes, file-like objects, and iterables. If no ``Content-Length`` nor ``Transfer-Encoding`` header field has been provided, :class:`HTTPHandler` will set these headers according to the type of *data*. ``Content-Length`` will be used to send bytes objects, while ``Transfer-Encoding: chunked`` as specified in :rfc:`7230`, Section 3.3.1 will be used to send files and other iterables. For an HTTP POST request method, *data* should be a buffer in the standard :mimetype:`application/x-www-form-urlencoded` format. The :func:`urllib.parse.urlencode` function takes a mapping or sequence of 2-tuples and returns an ASCII string in this format. It should be encoded to bytes before being used as the *data* parameter. *headers* should be a dictionary, and will be treated as if :meth:`add_header` was called with each key and value as arguments. This is often used to "spoof" the ``User-Agent`` header value, which is used by a browser to identify itself -- some HTTP servers only allow requests coming from common browsers as opposed to scripts. For example, Mozilla Firefox may identify itself as ``"Mozilla/5.0 (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11"``, while :mod:`urllib`'s default user agent string is ``"Python-urllib/2.6"`` (on Python 2.6). An appropriate ``Content-Type`` header should be included if the *data* argument is present. If this header has not been provided and *data* is not None, ``Content-Type: application/x-www-form-urlencoded`` will be added as a default. The final two arguments are only of interest for correct handling of third-party HTTP cookies: *origin_req_host* should be the request-host of the origin transaction, as defined by :rfc:`2965`. It defaults to ``http.cookiejar.request_host(self)``. This is the host name or IP address of the original request that was initiated by the user. For example, if the request is for an image in an HTML document, this should be the request-host of the request for the page containing the image. *unverifiable* should indicate whether the request is unverifiable, as defined by :rfc:`2965`. It defaults to ``False``. An unverifiable request is one whose URL the user did not have the option to approve. For example, if the request is for an image in an HTML document, and the user had no option to approve the automatic fetching of the image, this should be true. *method* should be a string that indicates the HTTP request method that will be used (e.g. ``'HEAD'``). If provided, its value is stored in the :attr:`~Request.method` attribute and is used by :meth:`get_method()`. The default is ``'GET'`` if *data* is ``None`` or ``'POST'`` otherwise. Subclasses may indicate a different default method by setting the :attr:`~Request.method` attribute in the class itself. .. note:: The request will not work as expected if the data object is unable to deliver its content more than once (e.g. a file or an iterable that can produce the content only once) and the request is retried for HTTP redirects or authentication. The *data* is sent to the HTTP server right away after the headers. There is no support for a 100-continue expectation in the library. .. versionchanged:: 3.3 :attr:`Request.method` argument is added to the Request class. .. versionchanged:: 3.4 Default :attr:`Request.method` may be indicated at the class level. .. versionchanged:: 3.6 Do not raise an error if the ``Content-Length`` has not been provided and *data* is neither ``None`` nor a bytes object. Fall back to use chunked transfer encoding instead. .. class:: OpenerDirector() The :class:`OpenerDirector` class opens URLs via :class:`BaseHandler`\ s chained together. It manages the chaining of handlers, and recovery from errors. .. class:: BaseHandler() This is the base class for all registered handlers --- and handles only the simple mechanics of registration. .. class:: HTTPDefaultErrorHandler() A class which defines a default handler for HTTP error responses; all responses are turned into :exc:`~urllib.error.HTTPError` exceptions. .. class:: HTTPRedirectHandler() A class to handle redirections. .. class:: HTTPCookieProcessor(cookiejar=None) A class to handle HTTP Cookies. .. class:: ProxyHandler(proxies=None) Cause requests to go through a proxy. If *proxies* is given, it must be a dictionary mapping protocol names to URLs of proxies. The default is to read the list of proxies from the environment variables :envvar:`_proxy`. If no proxy environment variables are set, then in a Windows environment proxy settings are obtained from the registry's Internet Settings section, and in a Mac OS X environment proxy information is retrieved from the OS X System Configuration Framework. To disable autodetected proxy pass an empty dictionary. The :envvar:`no_proxy` environment variable can be used to specify hosts which shouldn't be reached via proxy; if set, it should be a comma-separated list of hostname suffixes, optionally with ``:port`` appended, for example ``cern.ch,ncsa.uiuc.edu,some.host:8080``. .. note:: ``HTTP_PROXY`` will be ignored if a variable ``REQUEST_METHOD`` is set; see the documentation on :func:`~urllib.request.getproxies`. .. class:: HTTPPasswordMgr() Keep a database of ``(realm, uri) -> (user, password)`` mappings. .. class:: HTTPPasswordMgrWithDefaultRealm() Keep a database of ``(realm, uri) -> (user, password)`` mappings. A realm of ``None`` is considered a catch-all realm, which is searched if no other realm fits. .. class:: HTTPPasswordMgrWithPriorAuth() A variant of :class:`HTTPPasswordMgrWithDefaultRealm` that also has a database of ``uri -> is_authenticated`` mappings. Can be used by a BasicAuth handler to determine when to send authentication credentials immediately instead of waiting for a ``401`` response first. .. versionadded:: 3.5 .. class:: AbstractBasicAuthHandler(password_mgr=None) This is a mixin class that helps with HTTP authentication, both to the remote host and to a proxy. *password_mgr*, if given, should be something that is compatible with :class:`HTTPPasswordMgr`; refer to section :ref:`http-password-mgr` for information on the interface that must be supported. If *passwd_mgr* also provides ``is_authenticated`` and ``update_authenticated`` methods (see :ref:`http-password-mgr-with-prior-auth`), then the handler will use the ``is_authenticated`` result for a given URI to determine whether or not to send authentication credentials with the request. If ``is_authenticated`` returns ``True`` for the URI, credentials are sent. If ``is_authenticated`` is ``False``, credentials are not sent, and then if a ``401`` response is received the request is re-sent with the authentication credentials. If authentication succeeds, ``update_authenticated`` is called to set ``is_authenticated`` ``True`` for the URI, so that subsequent requests to the URI or any of its super-URIs will automatically include the authentication credentials. .. versionadded:: 3.5 Added ``is_authenticated`` support. .. class:: HTTPBasicAuthHandler(password_mgr=None) Handle authentication with the remote host. *password_mgr*, if given, should be something that is compatible with :class:`HTTPPasswordMgr`; refer to section :ref:`http-password-mgr` for information on the interface that must be supported. HTTPBasicAuthHandler will raise a :exc:`ValueError` when presented with a wrong Authentication scheme. .. class:: ProxyBasicAuthHandler(password_mgr=None) Handle authentication with the proxy. *password_mgr*, if given, should be something that is compatible with :class:`HTTPPasswordMgr`; refer to section :ref:`http-password-mgr` for information on the interface that must be supported. .. class:: AbstractDigestAuthHandler(password_mgr=None) This is a mixin class that helps with HTTP authentication, both to the remote host and to a proxy. *password_mgr*, if given, should be something that is compatible with :class:`HTTPPasswordMgr`; refer to section :ref:`http-password-mgr` for information on the interface that must be supported. .. class:: HTTPDigestAuthHandler(password_mgr=None) Handle authentication with the remote host. *password_mgr*, if given, should be something that is compatible with :class:`HTTPPasswordMgr`; refer to section :ref:`http-password-mgr` for information on the interface that must be supported. When both Digest Authentication Handler and Basic Authentication Handler are both added, Digest Authentication is always tried first. If the Digest Authentication returns a 40x response again, it is sent to Basic Authentication handler to Handle. This Handler method will raise a :exc:`ValueError` when presented with an authentication scheme other than Digest or Basic. .. versionchanged:: 3.3 Raise :exc:`ValueError` on unsupported Authentication Scheme. .. class:: ProxyDigestAuthHandler(password_mgr=None) Handle authentication with the proxy. *password_mgr*, if given, should be something that is compatible with :class:`HTTPPasswordMgr`; refer to section :ref:`http-password-mgr` for information on the interface that must be supported. .. class:: HTTPHandler() A class to handle opening of HTTP URLs. .. class:: HTTPSHandler(debuglevel=0, context=None, check_hostname=None) A class to handle opening of HTTPS URLs. *context* and *check_hostname* have the same meaning as in :class:`http.client.HTTPSConnection`. .. versionchanged:: 3.2 *context* and *check_hostname* were added. .. class:: FileHandler() Open local files. .. class:: DataHandler() Open data URLs. .. versionadded:: 3.4 .. class:: FTPHandler() Open FTP URLs. .. class:: CacheFTPHandler() Open FTP URLs, keeping a cache of open FTP connections to minimize delays. .. class:: UnknownHandler() A catch-all class to handle unknown URLs. .. class:: HTTPErrorProcessor() Process HTTP error responses. .. _request-objects: Request Objects --------------- The following methods describe :class:`Request`'s public interface, and so all may be overridden in subclasses. It also defines several public attributes that can be used by clients to inspect the parsed request. .. attribute:: Request.full_url The original URL passed to the constructor. .. versionchanged:: 3.4 Request.full_url is a property with setter, getter and a deleter. Getting :attr:`~Request.full_url` returns the original request URL with the fragment, if it was present. .. attribute:: Request.type The URI scheme. .. attribute:: Request.host The URI authority, typically a host, but may also contain a port separated by a colon. .. attribute:: Request.origin_req_host The original host for the request, without port. .. attribute:: Request.selector The URI path. If the :class:`Request` uses a proxy, then selector will be the full URL that is passed to the proxy. .. attribute:: Request.data The entity body for the request, or ``None`` if not specified. .. versionchanged:: 3.4 Changing value of :attr:`Request.data` now deletes "Content-Length" header if it was previously set or calculated. .. attribute:: Request.unverifiable boolean, indicates whether the request is unverifiable as defined by :rfc:`2965`. .. attribute:: Request.method The HTTP request method to use. By default its value is :const:`None`, which means that :meth:`~Request.get_method` will do its normal computation of the method to be used. Its value can be set (thus overriding the default computation in :meth:`~Request.get_method`) either by providing a default value by setting it at the class level in a :class:`Request` subclass, or by passing a value in to the :class:`Request` constructor via the *method* argument. .. versionadded:: 3.3 .. versionchanged:: 3.4 A default value can now be set in subclasses; previously it could only be set via the constructor argument. .. method:: Request.get_method() Return a string indicating the HTTP request method. If :attr:`Request.method` is not ``None``, return its value, otherwise return ``'GET'`` if :attr:`Request.data` is ``None``, or ``'POST'`` if it's not. This is only meaningful for HTTP requests. .. versionchanged:: 3.3 get_method now looks at the value of :attr:`Request.method`. .. method:: Request.add_header(key, val) Add another header to the request. Headers are currently ignored by all handlers except HTTP handlers, where they are added to the list of headers sent to the server. Note that there cannot be more than one header with the same name, and later calls will overwrite previous calls in case the *key* collides. Currently, this is no loss of HTTP functionality, since all headers which have meaning when used more than once have a (header-specific) way of gaining the same functionality using only one header. .. method:: Request.add_unredirected_header(key, header) Add a header that will not be added to a redirected request. .. method:: Request.has_header(header) Return whether the instance has the named header (checks both regular and unredirected). .. method:: Request.remove_header(header) Remove named header from the request instance (both from regular and unredirected headers). .. versionadded:: 3.4 .. method:: Request.get_full_url() Return the URL given in the constructor. .. versionchanged:: 3.4 Returns :attr:`Request.full_url` .. method:: Request.set_proxy(host, type) Prepare the request by connecting to a proxy server. The *host* and *type* will replace those of the instance, and the instance's selector will be the original URL given in the constructor. .. method:: Request.get_header(header_name, default=None) Return the value of the given header. If the header is not present, return the default value. .. method:: Request.header_items() Return a list of tuples (header_name, header_value) of the Request headers. .. versionchanged:: 3.4 The request methods add_data, has_data, get_data, get_type, get_host, get_selector, get_origin_req_host and is_unverifiable that were deprecated since 3.3 have been removed. .. _opener-director-objects: OpenerDirector Objects ---------------------- :class:`OpenerDirector` instances have the following methods: .. method:: OpenerDirector.add_handler(handler) *handler* should be an instance of :class:`BaseHandler`. The following methods are searched, and added to the possible chains (note that HTTP errors are a special case). * :meth:`protocol_open` --- signal that the handler knows how to open *protocol* URLs. * :meth:`http_error_type` --- signal that the handler knows how to handle HTTP errors with HTTP error code *type*. * :meth:`protocol_error` --- signal that the handler knows how to handle errors from (non-\ ``http``) *protocol*. * :meth:`protocol_request` --- signal that the handler knows how to pre-process *protocol* requests. * :meth:`protocol_response` --- signal that the handler knows how to post-process *protocol* responses. .. method:: OpenerDirector.open(url, data=None[, timeout]) Open the given *url* (which can be a request object or a string), optionally passing the given *data*. Arguments, return values and exceptions raised are the same as those of :func:`urlopen` (which simply calls the :meth:`open` method on the currently installed global :class:`OpenerDirector`). The optional *timeout* parameter specifies a timeout in seconds for blocking operations like the connection attempt (if not specified, the global default timeout setting will be used). The timeout feature actually works only for HTTP, HTTPS and FTP connections). .. method:: OpenerDirector.error(proto, *args) Handle an error of the given protocol. This will call the registered error handlers for the given protocol with the given arguments (which are protocol specific). The HTTP protocol is a special case which uses the HTTP response code to determine the specific error handler; refer to the :meth:`http_error_\*` methods of the handler classes. Return values and exceptions raised are the same as those of :func:`urlopen`. OpenerDirector objects open URLs in three stages: The order in which these methods are called within each stage is determined by sorting the handler instances. #. Every handler with a method named like :meth:`protocol_request` has that method called to pre-process the request. #. Handlers with a method named like :meth:`protocol_open` are called to handle the request. This stage ends when a handler either returns a non-\ :const:`None` value (ie. a response), or raises an exception (usually :exc:`~urllib.error.URLError`). Exceptions are allowed to propagate. In fact, the above algorithm is first tried for methods named :meth:`default_open`. If all such methods return :const:`None`, the algorithm is repeated for methods named like :meth:`protocol_open`. If all such methods return :const:`None`, the algorithm is repeated for methods named :meth:`unknown_open`. Note that the implementation of these methods may involve calls of the parent :class:`OpenerDirector` instance's :meth:`~OpenerDirector.open` and :meth:`~OpenerDirector.error` methods. #. Every handler with a method named like :meth:`protocol_response` has that method called to post-process the response. .. _base-handler-objects: BaseHandler Objects ------------------- :class:`BaseHandler` objects provide a couple of methods that are directly useful, and others that are meant to be used by derived classes. These are intended for direct use: .. method:: BaseHandler.add_parent(director) Add a director as parent. .. method:: BaseHandler.close() Remove any parents. The following attribute and methods should only be used by classes derived from :class:`BaseHandler`. .. note:: The convention has been adopted that subclasses defining :meth:`protocol_request` or :meth:`protocol_response` methods are named :class:`\*Processor`; all others are named :class:`\*Handler`. .. attribute:: BaseHandler.parent A valid :class:`OpenerDirector`, which can be used to open using a different protocol, or handle errors. .. method:: BaseHandler.default_open(req) This method is *not* defined in :class:`BaseHandler`, but subclasses should define it if they want to catch all URLs. This method, if implemented, will be called by the parent :class:`OpenerDirector`. It should return a file-like object as described in the return value of the :meth:`open` of :class:`OpenerDirector`, or ``None``. It should raise :exc:`~urllib.error.URLError`, unless a truly exceptional thing happens (for example, :exc:`MemoryError` should not be mapped to :exc:`URLError`). This method will be called before any protocol-specific open method. .. method:: BaseHandler.protocol_open(req) :noindex: This method is *not* defined in :class:`BaseHandler`, but subclasses should define it if they want to handle URLs with the given protocol. This method, if defined, will be called by the parent :class:`OpenerDirector`. Return values should be the same as for :meth:`default_open`. .. method:: BaseHandler.unknown_open(req) This method is *not* defined in :class:`BaseHandler`, but subclasses should define it if they want to catch all URLs with no specific registered handler to open it. This method, if implemented, will be called by the :attr:`parent` :class:`OpenerDirector`. Return values should be the same as for :meth:`default_open`. .. method:: BaseHandler.http_error_default(req, fp, code, msg, hdrs) This method is *not* defined in :class:`BaseHandler`, but subclasses should override it if they intend to provide a catch-all for otherwise unhandled HTTP errors. It will be called automatically by the :class:`OpenerDirector` getting the error, and should not normally be called in other circumstances. *req* will be a :class:`Request` object, *fp* will be a file-like object with the HTTP error body, *code* will be the three-digit code of the error, *msg* will be the user-visible explanation of the code and *hdrs* will be a mapping object with the headers of the error. Return values and exceptions raised should be the same as those of :func:`urlopen`. .. method:: BaseHandler.http_error_nnn(req, fp, code, msg, hdrs) *nnn* should be a three-digit HTTP error code. This method is also not defined in :class:`BaseHandler`, but will be called, if it exists, on an instance of a subclass, when an HTTP error with code *nnn* occurs. Subclasses should override this method to handle specific HTTP errors. Arguments, return values and exceptions raised should be the same as for :meth:`http_error_default`. .. method:: BaseHandler.protocol_request(req) :noindex: This method is *not* defined in :class:`BaseHandler`, but subclasses should define it if they want to pre-process requests of the given protocol. This method, if defined, will be called by the parent :class:`OpenerDirector`. *req* will be a :class:`Request` object. The return value should be a :class:`Request` object. .. method:: BaseHandler.protocol_response(req, response) :noindex: This method is *not* defined in :class:`BaseHandler`, but subclasses should define it if they want to post-process responses of the given protocol. This method, if defined, will be called by the parent :class:`OpenerDirector`. *req* will be a :class:`Request` object. *response* will be an object implementing the same interface as the return value of :func:`urlopen`. The return value should implement the same interface as the return value of :func:`urlopen`. .. _http-redirect-handler: HTTPRedirectHandler Objects --------------------------- .. note:: Some HTTP redirections require action from this module's client code. If this is the case, :exc:`~urllib.error.HTTPError` is raised. See :rfc:`2616` for details of the precise meanings of the various redirection codes. An :class:`HTTPError` exception raised as a security consideration if the HTTPRedirectHandler is presented with a redirected URL which is not an HTTP, HTTPS or FTP URL. .. method:: HTTPRedirectHandler.redirect_request(req, fp, code, msg, hdrs, newurl) Return a :class:`Request` or ``None`` in response to a redirect. This is called by the default implementations of the :meth:`http_error_30\*` methods when a redirection is received from the server. If a redirection should take place, return a new :class:`Request` to allow :meth:`http_error_30\*` to perform the redirect to *newurl*. Otherwise, raise :exc:`~urllib.error.HTTPError` if no other handler should try to handle this URL, or return ``None`` if you can't but another handler might. .. note:: The default implementation of this method does not strictly follow :rfc:`2616`, which says that 301 and 302 responses to ``POST`` requests must not be automatically redirected without confirmation by the user. In reality, browsers do allow automatic redirection of these responses, changing the POST to a ``GET``, and the default implementation reproduces this behavior. .. method:: HTTPRedirectHandler.http_error_301(req, fp, code, msg, hdrs) Redirect to the ``Location:`` or ``URI:`` URL. This method is called by the parent :class:`OpenerDirector` when getting an HTTP 'moved permanently' response. .. method:: HTTPRedirectHandler.http_error_302(req, fp, code, msg, hdrs) The same as :meth:`http_error_301`, but called for the 'found' response. .. method:: HTTPRedirectHandler.http_error_303(req, fp, code, msg, hdrs) The same as :meth:`http_error_301`, but called for the 'see other' response. .. method:: HTTPRedirectHandler.http_error_307(req, fp, code, msg, hdrs) The same as :meth:`http_error_301`, but called for the 'temporary redirect' response. .. _http-cookie-processor: HTTPCookieProcessor Objects --------------------------- :class:`HTTPCookieProcessor` instances have one attribute: .. attribute:: HTTPCookieProcessor.cookiejar The :class:`http.cookiejar.CookieJar` in which cookies are stored. .. _proxy-handler: ProxyHandler Objects -------------------- .. method:: ProxyHandler.protocol_open(request) :noindex: The :class:`ProxyHandler` will have a method :meth:`protocol_open` for every *protocol* which has a proxy in the *proxies* dictionary given in the constructor. The method will modify requests to go through the proxy, by calling ``request.set_proxy()``, and call the next handler in the chain to actually execute the protocol. .. _http-password-mgr: HTTPPasswordMgr Objects ----------------------- These methods are available on :class:`HTTPPasswordMgr` and :class:`HTTPPasswordMgrWithDefaultRealm` objects. .. method:: HTTPPasswordMgr.add_password(realm, uri, user, passwd) *uri* can be either a single URI, or a sequence of URIs. *realm*, *user* and *passwd* must be strings. This causes ``(user, passwd)`` to be used as authentication tokens when authentication for *realm* and a super-URI of any of the given URIs is given. .. method:: HTTPPasswordMgr.find_user_password(realm, authuri) Get user/password for given realm and URI, if any. This method will return ``(None, None)`` if there is no matching user/password. For :class:`HTTPPasswordMgrWithDefaultRealm` objects, the realm ``None`` will be searched if the given *realm* has no matching user/password. .. _http-password-mgr-with-prior-auth: HTTPPasswordMgrWithPriorAuth Objects ------------------------------------ This password manager extends :class:`HTTPPasswordMgrWithDefaultRealm` to support tracking URIs for which authentication credentials should always be sent. .. method:: HTTPPasswordMgrWithPriorAuth.add_password(realm, uri, user, \ passwd, is_authenticated=False) *realm*, *uri*, *user*, *passwd* are as for :meth:`HTTPPasswordMgr.add_password`. *is_authenticated* sets the initial value of the ``is_authenticated`` flag for the given URI or list of URIs. If *is_authenticated* is specified as ``True``, *realm* is ignored. .. method:: HTTPPasswordMgr.find_user_password(realm, authuri) Same as for :class:`HTTPPasswordMgrWithDefaultRealm` objects .. method:: HTTPPasswordMgrWithPriorAuth.update_authenticated(self, uri, \ is_authenticated=False) Update the ``is_authenticated`` flag for the given *uri* or list of URIs. .. method:: HTTPPasswordMgrWithPriorAuth.is_authenticated(self, authuri) Returns the current state of the ``is_authenticated`` flag for the given URI. .. _abstract-basic-auth-handler: AbstractBasicAuthHandler Objects -------------------------------- .. method:: AbstractBasicAuthHandler.http_error_auth_reqed(authreq, host, req, headers) Handle an authentication request by getting a user/password pair, and re-trying the request. *authreq* should be the name of the header where the information about the realm is included in the request, *host* specifies the URL and path to authenticate for, *req* should be the (failed) :class:`Request` object, and *headers* should be the error headers. *host* is either an authority (e.g. ``"python.org"``) or a URL containing an authority component (e.g. ``"http://python.org/"``). In either case, the authority must not contain a userinfo component (so, ``"python.org"`` and ``"python.org:80"`` are fine, ``"joe:password@python.org"`` is not). .. _http-basic-auth-handler: HTTPBasicAuthHandler Objects ---------------------------- .. method:: HTTPBasicAuthHandler.http_error_401(req, fp, code, msg, hdrs) Retry the request with authentication information, if available. .. _proxy-basic-auth-handler: ProxyBasicAuthHandler Objects ----------------------------- .. method:: ProxyBasicAuthHandler.http_error_407(req, fp, code, msg, hdrs) Retry the request with authentication information, if available. .. _abstract-digest-auth-handler: AbstractDigestAuthHandler Objects --------------------------------- .. method:: AbstractDigestAuthHandler.http_error_auth_reqed(authreq, host, req, headers) *authreq* should be the name of the header where the information about the realm is included in the request, *host* should be the host to authenticate to, *req* should be the (failed) :class:`Request` object, and *headers* should be the error headers. .. _http-digest-auth-handler: HTTPDigestAuthHandler Objects ----------------------------- .. method:: HTTPDigestAuthHandler.http_error_401(req, fp, code, msg, hdrs) Retry the request with authentication information, if available. .. _proxy-digest-auth-handler: ProxyDigestAuthHandler Objects ------------------------------ .. method:: ProxyDigestAuthHandler.http_error_407(req, fp, code, msg, hdrs) Retry the request with authentication information, if available. .. _http-handler-objects: HTTPHandler Objects ------------------- .. method:: HTTPHandler.http_open(req) Send an HTTP request, which can be either GET or POST, depending on ``req.has_data()``. .. _https-handler-objects: HTTPSHandler Objects -------------------- .. method:: HTTPSHandler.https_open(req) Send an HTTPS request, which can be either GET or POST, depending on ``req.has_data()``. .. _file-handler-objects: FileHandler Objects ------------------- .. method:: FileHandler.file_open(req) Open the file locally, if there is no host name, or the host name is ``'localhost'``. .. versionchanged:: 3.2 This method is applicable only for local hostnames. When a remote hostname is given, an :exc:`~urllib.error.URLError` is raised. .. _data-handler-objects: DataHandler Objects ------------------- .. method:: DataHandler.data_open(req) Read a data URL. This kind of URL contains the content encoded in the URL itself. The data URL syntax is specified in :rfc:`2397`. This implementation ignores white spaces in base64 encoded data URLs so the URL may be wrapped in whatever source file it comes from. But even though some browsers don't mind about a missing padding at the end of a base64 encoded data URL, this implementation will raise an :exc:`ValueError` in that case. .. _ftp-handler-objects: FTPHandler Objects ------------------ .. method:: FTPHandler.ftp_open(req) Open the FTP file indicated by *req*. The login is always done with empty username and password. .. _cacheftp-handler-objects: CacheFTPHandler Objects ----------------------- :class:`CacheFTPHandler` objects are :class:`FTPHandler` objects with the following additional methods: .. method:: CacheFTPHandler.setTimeout(t) Set timeout of connections to *t* seconds. .. method:: CacheFTPHandler.setMaxConns(m) Set maximum number of cached connections to *m*. .. _unknown-handler-objects: UnknownHandler Objects ---------------------- .. method:: UnknownHandler.unknown_open() Raise a :exc:`~urllib.error.URLError` exception. .. _http-error-processor-objects: HTTPErrorProcessor Objects -------------------------- .. method:: HTTPErrorProcessor.http_response(request, response) Process HTTP error responses. For 200 error codes, the response object is returned immediately. For non-200 error codes, this simply passes the job on to the :meth:`protocol_error_code` handler methods, via :meth:`OpenerDirector.error`. Eventually, :class:`HTTPDefaultErrorHandler` will raise an :exc:`~urllib.error.HTTPError` if no other handler handles the error. .. method:: HTTPErrorProcessor.https_response(request, response) Process HTTPS error responses. The behavior is same as :meth:`http_response`. .. _urllib-request-examples: Examples -------- In addition to the examples below, more examples are given in :ref:`urllib-howto`. This example gets the python.org main page and displays the first 300 bytes of it. :: >>> import urllib.request >>> with urllib.request.urlopen('http://www.python.org/') as f: ... print(f.read(300)) ... b'\n\n\n\n\n\n \n Python Programming ' Note that urlopen returns a bytes object. This is because there is no way for urlopen to automatically determine the encoding of the byte stream it receives from the HTTP server. In general, a program will decode the returned bytes object to string once it determines or guesses the appropriate encoding. The following W3C document, https://www.w3.org/International/O-charset\ , lists the various ways in which an (X)HTML or an XML document could have specified its encoding information. As the python.org website uses *utf-8* encoding as specified in its meta tag, we will use the same for decoding the bytes object. :: >>> with urllib.request.urlopen('http://www.python.org/') as f: ... print(f.read(100).decode('utf-8')) ... <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtm It is also possible to achieve the same result without using the :term:`context manager` approach. :: >>> import urllib.request >>> f = urllib.request.urlopen('http://www.python.org/') >>> print(f.read(100).decode('utf-8')) <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtm In the following example, we are sending a data-stream to the stdin of a CGI and reading the data it returns to us. Note that this example will only work when the Python installation supports SSL. :: >>> import urllib.request >>> req = urllib.request.Request(url='https://localhost/cgi-bin/test.cgi', ... data=b'This data is passed to stdin of the CGI') >>> with urllib.request.urlopen(req) as f: ... print(f.read().decode('utf-8')) ... Got Data: "This data is passed to stdin of the CGI" The code for the sample CGI used in the above example is:: #!/usr/bin/env python import sys data = sys.stdin.read() print('Content-type: text/plain\n\nGot Data: "%s"' % data) Here is an example of doing a ``PUT`` request using :class:`Request`:: import urllib.request DATA = b'some data' req = urllib.request.Request(url='http://localhost:8080', data=DATA,method='PUT') with urllib.request.urlopen(req) as f: pass print(f.status) print(f.reason) Use of Basic HTTP Authentication:: import urllib.request # Create an OpenerDirector with support for Basic HTTP Authentication... auth_handler = urllib.request.HTTPBasicAuthHandler() auth_handler.add_password(realm='PDQ Application', uri='https://mahler:8092/site-updates.py', user='klem', passwd='kadidd!ehopper') opener = urllib.request.build_opener(auth_handler) # ...and install it globally so it can be used with urlopen. urllib.request.install_opener(opener) urllib.request.urlopen('http://www.example.com/login.html') :func:`build_opener` provides many handlers by default, including a :class:`ProxyHandler`. By default, :class:`ProxyHandler` uses the environment variables named ``<scheme>_proxy``, where ``<scheme>`` is the URL scheme involved. For example, the :envvar:`http_proxy` environment variable is read to obtain the HTTP proxy's URL. This example replaces the default :class:`ProxyHandler` with one that uses programmatically-supplied proxy URLs, and adds proxy authorization support with :class:`ProxyBasicAuthHandler`. :: proxy_handler = urllib.request.ProxyHandler({'http': 'http://www.example.com:3128/'}) proxy_auth_handler = urllib.request.ProxyBasicAuthHandler() proxy_auth_handler.add_password('realm', 'host', 'username', 'password') opener = urllib.request.build_opener(proxy_handler, proxy_auth_handler) # This time, rather than install the OpenerDirector, we use it directly: opener.open('http://www.example.com/login.html') Adding HTTP headers: Use the *headers* argument to the :class:`Request` constructor, or:: import urllib.request req = urllib.request.Request('http://www.example.com/') req.add_header('Referer', 'http://www.python.org/') # Customize the default User-Agent header value: req.add_header('User-Agent', 'urllib-example/0.1 (Contact: . . .)') r = urllib.request.urlopen(req) :class:`OpenerDirector` automatically adds a :mailheader:`User-Agent` header to every :class:`Request`. To change this:: import urllib.request opener = urllib.request.build_opener() opener.addheaders = [('User-agent', 'Mozilla/5.0')] opener.open('http://www.example.com/') Also, remember that a few standard headers (:mailheader:`Content-Length`, :mailheader:`Content-Type` and :mailheader:`Host`) are added when the :class:`Request` is passed to :func:`urlopen` (or :meth:`OpenerDirector.open`). .. _urllib-examples: Here is an example session that uses the ``GET`` method to retrieve a URL containing parameters:: >>> import urllib.request >>> import urllib.parse >>> params = urllib.parse.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0}) >>> url = "http://www.musi-cal.com/cgi-bin/query?%s" % params >>> with urllib.request.urlopen(url) as f: ... print(f.read().decode('utf-8')) ... The following example uses the ``POST`` method instead. Note that params output from urlencode is encoded to bytes before it is sent to urlopen as data:: >>> import urllib.request >>> import urllib.parse >>> data = urllib.parse.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0}) >>> data = data.encode('ascii') >>> with urllib.request.urlopen("http://requestb.in/xrbl82xr", data) as f: ... print(f.read().decode('utf-8')) ... The following example uses an explicitly specified HTTP proxy, overriding environment settings:: >>> import urllib.request >>> proxies = {'http': 'http://proxy.example.com:8080/'} >>> opener = urllib.request.FancyURLopener(proxies) >>> with opener.open("http://www.python.org") as f: ... f.read().decode('utf-8') ... The following example uses no proxies at all, overriding environment settings:: >>> import urllib.request >>> opener = urllib.request.FancyURLopener({}) >>> with opener.open("http://www.python.org/") as f: ... f.read().decode('utf-8') ... Legacy interface ---------------- The following functions and classes are ported from the Python 2 module ``urllib`` (as opposed to ``urllib2``). They might become deprecated at some point in the future. .. function:: urlretrieve(url, filename=None, reporthook=None, data=None) Copy a network object denoted by a URL to a local file. If the URL points to a local file, the object will not be copied unless filename is supplied. Return a tuple ``(filename, headers)`` where *filename* is the local file name under which the object can be found, and *headers* is whatever the :meth:`info` method of the object returned by :func:`urlopen` returned (for a remote object). Exceptions are the same as for :func:`urlopen`. The second argument, if present, specifies the file location to copy to (if absent, the location will be a tempfile with a generated name). The third argument, if present, is a callable that will be called once on establishment of the network connection and once after each block read thereafter. The callable will be passed three arguments; a count of blocks transferred so far, a block size in bytes, and the total size of the file. The third argument may be ``-1`` on older FTP servers which do not return a file size in response to a retrieval request. The following example illustrates the most common usage scenario:: >>> import urllib.request >>> local_filename, headers = urllib.request.urlretrieve('http://python.org/') >>> html = open(local_filename) >>> html.close() If the *url* uses the :file:`http:` scheme identifier, the optional *data* argument may be given to specify a ``POST`` request (normally the request type is ``GET``). The *data* argument must be a bytes object in standard :mimetype:`application/x-www-form-urlencoded` format; see the :func:`urllib.parse.urlencode` function. :func:`urlretrieve` will raise :exc:`ContentTooShortError` when it detects that the amount of data available was less than the expected amount (which is the size reported by a *Content-Length* header). This can occur, for example, when the download is interrupted. The *Content-Length* is treated as a lower bound: if there's more data to read, urlretrieve reads more data, but if less data is available, it raises the exception. You can still retrieve the downloaded data in this case, it is stored in the :attr:`content` attribute of the exception instance. If no *Content-Length* header was supplied, urlretrieve can not check the size of the data it has downloaded, and just returns it. In this case you just have to assume that the download was successful. .. function:: urlcleanup() Cleans up temporary files that may have been left behind by previous calls to :func:`urlretrieve`. .. class:: URLopener(proxies=None, **x509) .. deprecated:: 3.3 Base class for opening and reading URLs. Unless you need to support opening objects using schemes other than :file:`http:`, :file:`ftp:`, or :file:`file:`, you probably want to use :class:`FancyURLopener`. By default, the :class:`URLopener` class sends a :mailheader:`User-Agent` header of ``urllib/VVV``, where *VVV* is the :mod:`urllib` version number. Applications can define their own :mailheader:`User-Agent` header by subclassing :class:`URLopener` or :class:`FancyURLopener` and setting the class attribute :attr:`version` to an appropriate string value in the subclass definition. The optional *proxies* parameter should be a dictionary mapping scheme names to proxy URLs, where an empty dictionary turns proxies off completely. Its default value is ``None``, in which case environmental proxy settings will be used if present, as discussed in the definition of :func:`urlopen`, above. Additional keyword parameters, collected in *x509*, may be used for authentication of the client when using the :file:`https:` scheme. The keywords *key_file* and *cert_file* are supported to provide an SSL key and certificate; both are needed to support client authentication. :class:`URLopener` objects will raise an :exc:`OSError` exception if the server returns an error code. .. method:: open(fullurl, data=None) Open *fullurl* using the appropriate protocol. This method sets up cache and proxy information, then calls the appropriate open method with its input arguments. If the scheme is not recognized, :meth:`open_unknown` is called. The *data* argument has the same meaning as the *data* argument of :func:`urlopen`. .. method:: open_unknown(fullurl, data=None) Overridable interface to open unknown URL types. .. method:: retrieve(url, filename=None, reporthook=None, data=None) Retrieves the contents of *url* and places it in *filename*. The return value is a tuple consisting of a local filename and either an :class:`email.message.Message` object containing the response headers (for remote URLs) or ``None`` (for local URLs). The caller must then open and read the contents of *filename*. If *filename* is not given and the URL refers to a local file, the input filename is returned. If the URL is non-local and *filename* is not given, the filename is the output of :func:`tempfile.mktemp` with a suffix that matches the suffix of the last path component of the input URL. If *reporthook* is given, it must be a function accepting three numeric parameters: A chunk number, the maximum size chunks are read in and the total size of the download (-1 if unknown). It will be called once at the start and after each chunk of data is read from the network. *reporthook* is ignored for local URLs. If the *url* uses the :file:`http:` scheme identifier, the optional *data* argument may be given to specify a ``POST`` request (normally the request type is ``GET``). The *data* argument must in standard :mimetype:`application/x-www-form-urlencoded` format; see the :func:`urllib.parse.urlencode` function. .. attribute:: version Variable that specifies the user agent of the opener object. To get :mod:`urllib` to tell servers that it is a particular user agent, set this in a subclass as a class variable or in the constructor before calling the base constructor. .. class:: FancyURLopener(...) .. deprecated:: 3.3 :class:`FancyURLopener` subclasses :class:`URLopener` providing default handling for the following HTTP response codes: 301, 302, 303, 307 and 401. For the 30x response codes listed above, the :mailheader:`Location` header is used to fetch the actual URL. For 401 response codes (authentication required), basic HTTP authentication is performed. For the 30x response codes, recursion is bounded by the value of the *maxtries* attribute, which defaults to 10. For all other response codes, the method :meth:`http_error_default` is called which you can override in subclasses to handle the error appropriately. .. note:: According to the letter of :rfc:`2616`, 301 and 302 responses to POST requests must not be automatically redirected without confirmation by the user. In reality, browsers do allow automatic redirection of these responses, changing the POST to a GET, and :mod:`urllib` reproduces this behaviour. The parameters to the constructor are the same as those for :class:`URLopener`. .. note:: When performing basic authentication, a :class:`FancyURLopener` instance calls its :meth:`prompt_user_passwd` method. The default implementation asks the users for the required information on the controlling terminal. A subclass may override this method to support more appropriate behavior if needed. The :class:`FancyURLopener` class offers one additional method that should be overloaded to provide the appropriate behavior: .. method:: prompt_user_passwd(host, realm) Return information needed to authenticate the user at the given host in the specified security realm. The return value should be a tuple, ``(user, password)``, which can be used for basic authentication. The implementation prompts for this information on the terminal; an application should override this method to use an appropriate interaction model in the local environment. :mod:`urllib.request` Restrictions ---------------------------------- .. index:: pair: HTTP; protocol pair: FTP; protocol * Currently, only the following protocols are supported: HTTP (versions 0.9 and 1.0), FTP, local files, and data URLs. .. versionchanged:: 3.4 Added support for data URLs. * The caching feature of :func:`urlretrieve` has been disabled until someone finds the time to hack proper processing of Expiration time headers. * There should be a function to query whether a particular URL is in the cache. * For backward compatibility, if a URL appears to point to a local file but the file can't be opened, the URL is re-interpreted using the FTP protocol. This can sometimes cause confusing error messages. * The :func:`urlopen` and :func:`urlretrieve` functions can cause arbitrarily long delays while waiting for a network connection to be set up. This means that it is difficult to build an interactive Web client using these functions without using threads. .. index:: single: HTML pair: HTTP; protocol * The data returned by :func:`urlopen` or :func:`urlretrieve` is the raw data returned by the server. This may be binary data (such as an image), plain text or (for example) HTML. The HTTP protocol provides type information in the reply header, which can be inspected by looking at the :mailheader:`Content-Type` header. If the returned data is HTML, you can use the module :mod:`html.parser` to parse it. .. index:: single: FTP * The code handling the FTP protocol cannot differentiate between a file and a directory. This can lead to unexpected behavior when attempting to read a URL that points to a file that is not accessible. If the URL ends in a ``/``, it is assumed to refer to a directory and will be handled accordingly. But if an attempt to read a file leads to a 550 error (meaning the URL cannot be found or is not accessible, often for permission reasons), then the path is treated as a directory in order to handle the case when a directory is specified by a URL but the trailing ``/`` has been left off. This can cause misleading results when you try to fetch a file whose read permissions make it inaccessible; the FTP code will try to read it, fail with a 550 error, and then perform a directory listing for the unreadable file. If fine-grained control is needed, consider using the :mod:`ftplib` module, subclassing :class:`FancyURLopener`, or changing *_urlopener* to meet your needs. :mod:`urllib.response` --- Response classes used by urllib ========================================================== .. module:: urllib.response :synopsis: Response classes used by urllib. The :mod:`urllib.response` module defines functions and classes which define a minimal file like interface, including ``read()`` and ``readline()``. The typical response object is an addinfourl instance, which defines an ``info()`` method and that returns headers and a ``geturl()`` method that returns the url. Functions defined by this module are used internally by the :mod:`urllib.request` module. PK����� 3].�MO"��"����library/curses.ascii.rst.txtnu�[��������:mod:`curses.ascii` --- Utilities for ASCII characters ====================================================== .. module:: curses.ascii :synopsis: Constants and set-membership functions for ASCII characters. .. moduleauthor:: Eric S. Raymond <esr@thyrsus.com> .. sectionauthor:: Eric S. Raymond <esr@thyrsus.com> -------------- The :mod:`curses.ascii` module supplies name constants for ASCII characters and functions to test membership in various ASCII character classes. The constants supplied are names for control characters as follows: +--------------+----------------------------------------------+ | Name | Meaning | +==============+==============================================+ | :const:`NUL` | | +--------------+----------------------------------------------+ | :const:`SOH` | Start of heading, console interrupt | +--------------+----------------------------------------------+ | :const:`STX` | Start of text | +--------------+----------------------------------------------+ | :const:`ETX` | End of text | +--------------+----------------------------------------------+ | :const:`EOT` | End of transmission | +--------------+----------------------------------------------+ | :const:`ENQ` | Enquiry, goes with :const:`ACK` flow control | +--------------+----------------------------------------------+ | :const:`ACK` | Acknowledgement | +--------------+----------------------------------------------+ | :const:`BEL` | Bell | +--------------+----------------------------------------------+ | :const:`BS` | Backspace | +--------------+----------------------------------------------+ | :const:`TAB` | Tab | +--------------+----------------------------------------------+ | :const:`HT` | Alias for :const:`TAB`: "Horizontal tab" | +--------------+----------------------------------------------+ | :const:`LF` | Line feed | +--------------+----------------------------------------------+ | :const:`NL` | Alias for :const:`LF`: "New line" | +--------------+----------------------------------------------+ | :const:`VT` | Vertical tab | +--------------+----------------------------------------------+ | :const:`FF` | Form feed | +--------------+----------------------------------------------+ | :const:`CR` | Carriage return | +--------------+----------------------------------------------+ | :const:`SO` | Shift-out, begin alternate character set | +--------------+----------------------------------------------+ | :const:`SI` | Shift-in, resume default character set | +--------------+----------------------------------------------+ | :const:`DLE` | Data-link escape | +--------------+----------------------------------------------+ | :const:`DC1` | XON, for flow control | +--------------+----------------------------------------------+ | :const:`DC2` | Device control 2, block-mode flow control | +--------------+----------------------------------------------+ | :const:`DC3` | XOFF, for flow control | +--------------+----------------------------------------------+ | :const:`DC4` | Device control 4 | +--------------+----------------------------------------------+ | :const:`NAK` | Negative acknowledgement | +--------------+----------------------------------------------+ | :const:`SYN` | Synchronous idle | +--------------+----------------------------------------------+ | :const:`ETB` | End transmission block | +--------------+----------------------------------------------+ | :const:`CAN` | Cancel | +--------------+----------------------------------------------+ | :const:`EM` | End of medium | +--------------+----------------------------------------------+ | :const:`SUB` | Substitute | +--------------+----------------------------------------------+ | :const:`ESC` | Escape | +--------------+----------------------------------------------+ | :const:`FS` | File separator | +--------------+----------------------------------------------+ | :const:`GS` | Group separator | +--------------+----------------------------------------------+ | :const:`RS` | Record separator, block-mode terminator | +--------------+----------------------------------------------+ | :const:`US` | Unit separator | +--------------+----------------------------------------------+ | :const:`SP` | Space | +--------------+----------------------------------------------+ | :const:`DEL` | Delete | +--------------+----------------------------------------------+ Note that many of these have little practical significance in modern usage. The mnemonics derive from teleprinter conventions that predate digital computers. The module supplies the following functions, patterned on those in the standard C library: .. function:: isalnum(c) Checks for an ASCII alphanumeric character; it is equivalent to ``isalpha(c) or isdigit(c)``. .. function:: isalpha(c) Checks for an ASCII alphabetic character; it is equivalent to ``isupper(c) or islower(c)``. .. function:: isascii(c) Checks for a character value that fits in the 7-bit ASCII set. .. function:: isblank(c) Checks for an ASCII whitespace character; space or horizontal tab. .. function:: iscntrl(c) Checks for an ASCII control character (in the range 0x00 to 0x1f or 0x7f). .. function:: isdigit(c) Checks for an ASCII decimal digit, ``'0'`` through ``'9'``. This is equivalent to ``c in string.digits``. .. function:: isgraph(c) Checks for ASCII any printable character except space. .. function:: islower(c) Checks for an ASCII lower-case character. .. function:: isprint(c) Checks for any ASCII printable character including space. .. function:: ispunct(c) Checks for any printable ASCII character which is not a space or an alphanumeric character. .. function:: isspace(c) Checks for ASCII white-space characters; space, line feed, carriage return, form feed, horizontal tab, vertical tab. .. function:: isupper(c) Checks for an ASCII uppercase letter. .. function:: isxdigit(c) Checks for an ASCII hexadecimal digit. This is equivalent to ``c in string.hexdigits``. .. function:: isctrl(c) Checks for an ASCII control character (ordinal values 0 to 31). .. function:: ismeta(c) Checks for a non-ASCII character (ordinal values 0x80 and above). These functions accept either integers or single-character strings; when the argument is a string, it is first converted using the built-in function :func:`ord`. Note that all these functions check ordinal bit values derived from the character of the string you pass in; they do not actually know anything about the host machine's character encoding. The following two functions take either a single-character string or integer byte value; they return a value of the same type. .. function:: ascii(c) Return the ASCII value corresponding to the low 7 bits of *c*. .. function:: ctrl(c) Return the control character corresponding to the given character (the character bit value is bitwise-anded with 0x1f). .. function:: alt(c) Return the 8-bit character corresponding to the given ASCII character (the character bit value is bitwise-ored with 0x80). The following function takes either a single-character string or integer value; it returns a string. .. function:: unctrl(c) Return a string representation of the ASCII character *c*. If *c* is printable, this string is the character itself. If the character is a control character (0x00--0x1f) the string consists of a caret (``'^'``) followed by the corresponding uppercase letter. If the character is an ASCII delete (0x7f) the string is ``'^?'``. If the character has its meta bit (0x80) set, the meta bit is stripped, the preceding rules applied, and ``'!'`` prepended to the result. .. data:: controlnames A 33-element string array that contains the ASCII mnemonics for the thirty-two ASCII control characters from 0 (NUL) to 0x1f (US), in order, plus the mnemonic ``SP`` for the space character. PK����� 3]u �� ����library/othergui.rst.txtnu�[��������.. _other-gui-packages: Other Graphical User Interface Packages ======================================= Major cross-platform (Windows, Mac OS X, Unix-like) GUI toolkits are available for Python: .. seealso:: `PyGObject <https://wiki.gnome.org/Projects/PyGObject>`_ PyGObject provides introspection bindings for C libraries using `GObject <https://developer.gnome.org/gobject/stable/>`_. One of these libraries is the `GTK+ 3 <http://www.gtk.org/>`_ widget set. GTK+ comes with many more widgets than Tkinter provides. An online `Python GTK+ 3 Tutorial <https://python-gtk-3-tutorial.readthedocs.org/en/latest/>`_ is available. `PyGTK <http://www.pygtk.org/>`_ PyGTK provides bindings for an older version of the library, GTK+ 2. It provides an object oriented interface that is slightly higher level than the C one. There are also bindings to `GNOME <https://www.gnome.org/>`_. An online `tutorial <http://www.pygtk.org/pygtk2tutorial/index.html>`_ is available. `PyQt <https://riverbankcomputing.com/software/pyqt/intro>`_ PyQt is a :program:`sip`\ -wrapped binding to the Qt toolkit. Qt is an extensive C++ GUI application development framework that is available for Unix, Windows and Mac OS X. :program:`sip` is a tool for generating bindings for C++ libraries as Python classes, and is specifically designed for Python. `PySide <https://wiki.qt.io/PySide>`_ PySide is a newer binding to the Qt toolkit, provided by Nokia. Compared to PyQt, its licensing scheme is friendlier to non-open source applications. `wxPython <http://www.wxpython.org>`_ wxPython is a cross-platform GUI toolkit for Python that is built around the popular `wxWidgets <https://www.wxwidgets.org/>`_ (formerly wxWindows) C++ toolkit. It provides a native look and feel for applications on Windows, Mac OS X, and Unix systems by using each platform's native widgets where ever possible, (GTK+ on Unix-like systems). In addition to an extensive set of widgets, wxPython provides classes for online documentation and context sensitive help, printing, HTML viewing, low-level device context drawing, drag and drop, system clipboard access, an XML-based resource format and more, including an ever growing library of user-contributed modules. PyGTK, PyQt, and wxPython, all have a modern look and feel and more widgets than Tkinter. In addition, there are many other GUI toolkits for Python, both cross-platform, and platform-specific. See the `GUI Programming <https://wiki.python.org/moin/GuiProgramming>`_ page in the Python Wiki for a much more complete list, and also for links to documents where the different GUI toolkits are compared. PK����� 3]_AԘ#��#��$��library/email.contentmanager.rst.txtnu�[��������:mod:`email.contentmanager`: Managing MIME Content -------------------------------------------------- .. module:: email.contentmanager :synopsis: Storing and Retrieving Content from MIME Parts .. moduleauthor:: R. David Murray <rdmurray@bitdance.com> .. sectionauthor:: R. David Murray <rdmurray@bitdance.com> **Source code:** :source:`Lib/email/contentmanager.py` ------------ .. versionadded:: 3.6 [1]_ .. class:: ContentManager() Base class for content managers. Provides the standard registry mechanisms to register converters between MIME content and other representations, as well as the ``get_content`` and ``set_content`` dispatch methods. .. method:: get_content(msg, *args, **kw) Look up a handler function based on the ``mimetype`` of *msg* (see next paragraph), call it, passing through all arguments, and return the result of the call. The expectation is that the handler will extract the payload from *msg* and return an object that encodes information about the extracted data. To find the handler, look for the following keys in the registry, stopping with the first one found: * the string representing the full MIME type (``maintype/subtype``) * the string representing the ``maintype`` * the empty string If none of these keys produce a handler, raise a :exc:`KeyError` for the full MIME type. .. method:: set_content(msg, obj, *args, **kw) If the ``maintype`` is ``multipart``, raise a :exc:`TypeError`; otherwise look up a handler function based on the type of *obj* (see next paragraph), call :meth:`~email.message.EmailMessage.clear_content` on the *msg*, and call the handler function, passing through all arguments. The expectation is that the handler will transform and store *obj* into *msg*, possibly making other changes to *msg* as well, such as adding various MIME headers to encode information needed to interpret the stored data. To find the handler, obtain the type of *obj* (``typ = type(obj)``), and look for the following keys in the registry, stopping with the first one found: * the type itself (``typ``) * the type's fully qualified name (``typ.__module__ + '.' + typ.__qualname__``). * the type's qualname (``typ.__qualname__``) * the type's name (``typ.__name__``). If none of the above match, repeat all of the checks above for each of the types in the :term:`MRO` (``typ.__mro__``). Finally, if no other key yields a handler, check for a handler for the key ``None``. If there is no handler for ``None``, raise a :exc:`KeyError` for the fully qualified name of the type. Also add a :mailheader:`MIME-Version` header if one is not present (see also :class:`.MIMEPart`). .. method:: add_get_handler(key, handler) Record the function *handler* as the handler for *key*. For the possible values of *key*, see :meth:`get_content`. .. method:: add_set_handler(typekey, handler) Record *handler* as the function to call when an object of a type matching *typekey* is passed to :meth:`set_content`. For the possible values of *typekey*, see :meth:`set_content`. Content Manager Instances ~~~~~~~~~~~~~~~~~~~~~~~~~ Currently the email package provides only one concrete content manager, :data:`raw_data_manager`, although more may be added in the future. :data:`raw_data_manager` is the :attr:`~email.policy.EmailPolicy.content_manager` provided by :attr:`~email.policy.EmailPolicy` and its derivatives. .. data:: raw_data_manager This content manager provides only a minimum interface beyond that provided by :class:`~email.message.Message` itself: it deals only with text, raw byte strings, and :class:`~email.message.Message` objects. Nevertheless, it provides significant advantages compared to the base API: ``get_content`` on a text part will return a unicode string without the application needing to manually decode it, ``set_content`` provides a rich set of options for controlling the headers added to a part and controlling the content transfer encoding, and it enables the use of the various ``add_`` methods, thereby simplifying the creation of multipart messages. .. method:: get_content(msg, errors='replace') Return the payload of the part as either a string (for ``text`` parts), an :class:`~email.message.EmailMessage` object (for ``message/rfc822`` parts), or a ``bytes`` object (for all other non-multipart types). Raise a :exc:`KeyError` if called on a ``multipart``. If the part is a ``text`` part and *errors* is specified, use it as the error handler when decoding the payload to unicode. The default error handler is ``replace``. .. method:: set_content(msg, <'str'>, subtype="plain", charset='utf-8' \ cte=None, \ disposition=None, filename=None, cid=None, \ params=None, headers=None) set_content(msg, <'bytes'>, maintype, subtype, cte="base64", \ disposition=None, filename=None, cid=None, \ params=None, headers=None) set_content(msg, <'EmailMessage'>, cte=None, \ disposition=None, filename=None, cid=None, \ params=None, headers=None) Add headers and payload to *msg*: Add a :mailheader:`Content-Type` header with a ``maintype/subtype`` value. * For ``str``, set the MIME ``maintype`` to ``text``, and set the subtype to *subtype* if it is specified, or ``plain`` if it is not. * For ``bytes``, use the specified *maintype* and *subtype*, or raise a :exc:`TypeError` if they are not specified. * For :class:`~email.message.EmailMessage` objects, set the maintype to ``message``, and set the subtype to *subtype* if it is specified or ``rfc822`` if it is not. If *subtype* is ``partial``, raise an error (``bytes`` objects must be used to construct ``message/partial`` parts). If *charset* is provided (which is valid only for ``str``), encode the string to bytes using the specified character set. The default is ``utf-8``. If the specified *charset* is a known alias for a standard MIME charset name, use the standard charset instead. If *cte* is set, encode the payload using the specified content transfer encoding, and set the :mailheader:`Content-Transfer-Encoding` header to that value. Possible values for *cte* are ``quoted-printable``, ``base64``, ``7bit``, ``8bit``, and ``binary``. If the input cannot be encoded in the specified encoding (for example, specifying a *cte* of ``7bit`` for an input that contains non-ASCII values), raise a :exc:`ValueError`. * For ``str`` objects, if *cte* is not set use heuristics to determine the most compact encoding. * For :class:`~email.message.EmailMessage`, per :rfc:`2046`, raise an error if a *cte* of ``quoted-printable`` or ``base64`` is requested for *subtype* ``rfc822``, and for any *cte* other than ``7bit`` for *subtype* ``external-body``. For ``message/rfc822``, use ``8bit`` if *cte* is not specified. For all other values of *subtype*, use ``7bit``. .. note:: A *cte* of ``binary`` does not actually work correctly yet. The ``EmailMessage`` object as modified by ``set_content`` is correct, but :class:`~email.generator.BytesGenerator` does not serialize it correctly. If *disposition* is set, use it as the value of the :mailheader:`Content-Disposition` header. If not specified, and *filename* is specified, add the header with the value ``attachment``. If *disposition* is not specified and *filename* is also not specified, do not add the header. The only valid values for *disposition* are ``attachment`` and ``inline``. If *filename* is specified, use it as the value of the ``filename`` parameter of the :mailheader:`Content-Disposition` header. If *cid* is specified, add a :mailheader:`Content-ID` header with *cid* as its value. If *params* is specified, iterate its ``items`` method and use the resulting ``(key, value)`` pairs to set additional parameters on the :mailheader:`Content-Type` header. If *headers* is specified and is a list of strings of the form ``headername: headervalue`` or a list of ``header`` objects (distinguished from strings by having a ``name`` attribute), add the headers to *msg*. .. rubric:: Footnotes .. [1] Originally added in 3.4 as a :term:`provisional module <provisional package>` PK����� 3]{mØ%��%����library/platform.rst.txtnu�[��������:mod:`platform` --- Access to underlying platform's identifying data ===================================================================== .. module:: platform :synopsis: Retrieves as much platform identifying data as possible. .. moduleauthor:: Marc-André Lemburg <mal@egenix.com> .. sectionauthor:: Bjorn Pettersen <bpettersen@corp.fairisaac.com> **Source code:** :source:`Lib/platform.py` -------------- .. note:: Specific platforms listed alphabetically, with Linux included in the Unix section. Cross Platform -------------- .. function:: architecture(executable=sys.executable, bits='', linkage='') Queries the given executable (defaults to the Python interpreter binary) for various architecture information. Returns a tuple ``(bits, linkage)`` which contain information about the bit architecture and the linkage format used for the executable. Both values are returned as strings. Values that cannot be determined are returned as given by the parameter presets. If bits is given as ``''``, the ``sizeof(pointer)`` (or ``sizeof(long)`` on Python version < 1.5.2) is used as indicator for the supported pointer size. The function relies on the system's :file:`file` command to do the actual work. This is available on most if not all Unix platforms and some non-Unix platforms and then only if the executable points to the Python interpreter. Reasonable defaults are used when the above needs are not met. .. note:: On Mac OS X (and perhaps other platforms), executable files may be universal files containing multiple architectures. To get at the "64-bitness" of the current interpreter, it is more reliable to query the :attr:`sys.maxsize` attribute:: is_64bits = sys.maxsize > 2**32 .. function:: machine() Returns the machine type, e.g. ``'i386'``. An empty string is returned if the value cannot be determined. .. function:: node() Returns the computer's network name (may not be fully qualified!). An empty string is returned if the value cannot be determined. .. function:: platform(aliased=0, terse=0) Returns a single string identifying the underlying platform with as much useful information as possible. The output is intended to be *human readable* rather than machine parseable. It may look different on different platforms and this is intended. If *aliased* is true, the function will use aliases for various platforms that report system names which differ from their common names, for example SunOS will be reported as Solaris. The :func:`system_alias` function is used to implement this. Setting *terse* to true causes the function to return only the absolute minimum information needed to identify the platform. .. function:: processor() Returns the (real) processor name, e.g. ``'amdk6'``. An empty string is returned if the value cannot be determined. Note that many platforms do not provide this information or simply return the same value as for :func:`machine`. NetBSD does this. .. function:: python_build() Returns a tuple ``(buildno, builddate)`` stating the Python build number and date as strings. .. function:: python_compiler() Returns a string identifying the compiler used for compiling Python. .. function:: python_branch() Returns a string identifying the Python implementation SCM branch. .. function:: python_implementation() Returns a string identifying the Python implementation. Possible return values are: 'CPython', 'IronPython', 'Jython', 'PyPy'. .. function:: python_revision() Returns a string identifying the Python implementation SCM revision. .. function:: python_version() Returns the Python version as string ``'major.minor.patchlevel'``. Note that unlike the Python ``sys.version``, the returned value will always include the patchlevel (it defaults to 0). .. function:: python_version_tuple() Returns the Python version as tuple ``(major, minor, patchlevel)`` of strings. Note that unlike the Python ``sys.version``, the returned value will always include the patchlevel (it defaults to ``'0'``). .. function:: release() Returns the system's release, e.g. ``'2.2.0'`` or ``'NT'`` An empty string is returned if the value cannot be determined. .. function:: system() Returns the system/OS name, e.g. ``'Linux'``, ``'Windows'``, or ``'Java'``. An empty string is returned if the value cannot be determined. .. function:: system_alias(system, release, version) Returns ``(system, release, version)`` aliased to common marketing names used for some systems. It also does some reordering of the information in some cases where it would otherwise cause confusion. .. function:: version() Returns the system's release version, e.g. ``'#3 on degas'``. An empty string is returned if the value cannot be determined. .. function:: uname() Fairly portable uname interface. Returns a :func:`~collections.namedtuple` containing six attributes: :attr:`system`, :attr:`node`, :attr:`release`, :attr:`version`, :attr:`machine`, and :attr:`processor`. Note that this adds a sixth attribute (:attr:`processor`) not present in the :func:`os.uname` result. Also, the attribute names are different for the first two attributes; :func:`os.uname` names them :attr:`sysname` and :attr:`nodename`. Entries which cannot be determined are set to ``''``. .. versionchanged:: 3.3 Result changed from a tuple to a namedtuple. Java Platform ------------- .. function:: java_ver(release='', vendor='', vminfo=('','',''), osinfo=('','','')) Version interface for Jython. Returns a tuple ``(release, vendor, vminfo, osinfo)`` with *vminfo* being a tuple ``(vm_name, vm_release, vm_vendor)`` and *osinfo* being a tuple ``(os_name, os_version, os_arch)``. Values which cannot be determined are set to the defaults given as parameters (which all default to ``''``). Windows Platform ---------------- .. function:: win32_ver(release='', version='', csd='', ptype='') Get additional version information from the Windows Registry and return a tuple ``(release, version, csd, ptype)`` referring to OS release, version number, CSD level (service pack) and OS type (multi/single processor). As a hint: *ptype* is ``'Uniprocessor Free'`` on single processor NT machines and ``'Multiprocessor Free'`` on multi processor machines. The *'Free'* refers to the OS version being free of debugging code. It could also state *'Checked'* which means the OS version uses debugging code, i.e. code that checks arguments, ranges, etc. .. note:: This function works best with Mark Hammond's :mod:`win32all` package installed, but also on Python 2.3 and later (support for this was added in Python 2.6). It obviously only runs on Win32 compatible platforms. Win95/98 specific ^^^^^^^^^^^^^^^^^ .. function:: popen(cmd, mode='r', bufsize=-1) Portable :func:`popen` interface. Find a working popen implementation preferring :func:`win32pipe.popen`. On Windows NT, :func:`win32pipe.popen` should work; on Windows 9x it hangs due to bugs in the MS C library. .. deprecated:: 3.3 This function is obsolete. Use the :mod:`subprocess` module. Check especially the :ref:`subprocess-replacements` section. Mac OS Platform --------------- .. function:: mac_ver(release='', versioninfo=('','',''), machine='') Get Mac OS version information and return it as tuple ``(release, versioninfo, machine)`` with *versioninfo* being a tuple ``(version, dev_stage, non_release_version)``. Entries which cannot be determined are set to ``''``. All tuple entries are strings. Unix Platforms -------------- .. function:: dist(distname='', version='', id='', supported_dists=('SuSE','debian','redhat','mandrake',...)) This is another name for :func:`linux_distribution`. .. deprecated-removed:: 3.5 3.8 See alternative like the `distro <https://pypi.org/project/distro>`_ package. .. function:: linux_distribution(distname='', version='', id='', supported_dists=('SuSE','debian','redhat','mandrake',...), full_distribution_name=1) Tries to determine the name of the Linux OS distribution name. ``supported_dists`` may be given to define the set of Linux distributions to look for. It defaults to a list of currently supported Linux distributions identified by their release file name. If ``full_distribution_name`` is true (default), the full distribution read from the OS is returned. Otherwise the short name taken from ``supported_dists`` is used. Returns a tuple ``(distname,version,id)`` which defaults to the args given as parameters. ``id`` is the item in parentheses after the version number. It is usually the version codename. .. deprecated-removed:: 3.5 3.8 See alternative like the `distro <https://pypi.org/project/distro>`_ package. .. function:: libc_ver(executable=sys.executable, lib='', version='', chunksize=16384) Tries to determine the libc version against which the file executable (defaults to the Python interpreter) is linked. Returns a tuple of strings ``(lib, version)`` which default to the given parameters in case the lookup fails. Note that this function has intimate knowledge of how different libc versions add symbols to the executable is probably only usable for executables compiled using :program:`gcc`. The file is read and scanned in chunks of *chunksize* bytes. PK����� 3]͍K"��"����library/pkgutil.rst.txtnu�[��������:mod:`pkgutil` --- Package extension utility ============================================ .. module:: pkgutil :synopsis: Utilities for the import system. **Source code:** :source:`Lib/pkgutil.py` -------------- This module provides utilities for the import system, in particular package support. .. class:: ModuleInfo(module_finder, name, ispkg) A namedtuple that holds a brief summary of a module's info. .. versionadded:: 3.6 .. function:: extend_path(path, name) Extend the search path for the modules which comprise a package. Intended use is to place the following code in a package's :file:`__init__.py`:: from pkgutil import extend_path __path__ = extend_path(__path__, __name__) This will add to the package's ``__path__`` all subdirectories of directories on ``sys.path`` named after the package. This is useful if one wants to distribute different parts of a single logical package as multiple directories. It also looks for :file:`\*.pkg` files beginning where ``*`` matches the *name* argument. This feature is similar to :file:`\*.pth` files (see the :mod:`site` module for more information), except that it doesn't special-case lines starting with ``import``. A :file:`\*.pkg` file is trusted at face value: apart from checking for duplicates, all entries found in a :file:`\*.pkg` file are added to the path, regardless of whether they exist on the filesystem. (This is a feature.) If the input path is not a list (as is the case for frozen packages) it is returned unchanged. The input path is not modified; an extended copy is returned. Items are only appended to the copy at the end. It is assumed that :data:`sys.path` is a sequence. Items of :data:`sys.path` that are not strings referring to existing directories are ignored. Unicode items on :data:`sys.path` that cause errors when used as filenames may cause this function to raise an exception (in line with :func:`os.path.isdir` behavior). .. class:: ImpImporter(dirname=None) :pep:`302` Finder that wraps Python's "classic" import algorithm. If *dirname* is a string, a :pep:`302` finder is created that searches that directory. If *dirname* is ``None``, a :pep:`302` finder is created that searches the current :data:`sys.path`, plus any modules that are frozen or built-in. Note that :class:`ImpImporter` does not currently support being used by placement on :data:`sys.meta_path`. .. deprecated:: 3.3 This emulation is no longer needed, as the standard import mechanism is now fully PEP 302 compliant and available in :mod:`importlib`. .. class:: ImpLoader(fullname, file, filename, etc) :term:`Loader` that wraps Python's "classic" import algorithm. .. deprecated:: 3.3 This emulation is no longer needed, as the standard import mechanism is now fully PEP 302 compliant and available in :mod:`importlib`. .. function:: find_loader(fullname) Retrieve a module :term:`loader` for the given *fullname*. This is a backwards compatibility wrapper around :func:`importlib.util.find_spec` that converts most failures to :exc:`ImportError` and only returns the loader rather than the full :class:`ModuleSpec`. .. versionchanged:: 3.3 Updated to be based directly on :mod:`importlib` rather than relying on the package internal PEP 302 import emulation. .. versionchanged:: 3.4 Updated to be based on :pep:`451` .. function:: get_importer(path_item) Retrieve a :term:`finder` for the given *path_item*. The returned finder is cached in :data:`sys.path_importer_cache` if it was newly created by a path hook. The cache (or part of it) can be cleared manually if a rescan of :data:`sys.path_hooks` is necessary. .. versionchanged:: 3.3 Updated to be based directly on :mod:`importlib` rather than relying on the package internal PEP 302 import emulation. .. function:: get_loader(module_or_name) Get a :term:`loader` object for *module_or_name*. If the module or package is accessible via the normal import mechanism, a wrapper around the relevant part of that machinery is returned. Returns ``None`` if the module cannot be found or imported. If the named module is not already imported, its containing package (if any) is imported, in order to establish the package ``__path__``. .. versionchanged:: 3.3 Updated to be based directly on :mod:`importlib` rather than relying on the package internal PEP 302 import emulation. .. versionchanged:: 3.4 Updated to be based on :pep:`451` .. function:: iter_importers(fullname='') Yield :term:`finder` objects for the given module name. If fullname contains a '.', the finders will be for the package containing fullname, otherwise they will be all registered top level finders (i.e. those on both sys.meta_path and sys.path_hooks). If the named module is in a package, that package is imported as a side effect of invoking this function. If no module name is specified, all top level finders are produced. .. versionchanged:: 3.3 Updated to be based directly on :mod:`importlib` rather than relying on the package internal PEP 302 import emulation. .. function:: iter_modules(path=None, prefix='') Yields :class:`ModuleInfo` for all submodules on *path*, or, if *path* is ``None``, all top-level modules on ``sys.path``. *path* should be either ``None`` or a list of paths to look for modules in. *prefix* is a string to output on the front of every module name on output. .. note:: Only works for a :term:`finder` which defines an ``iter_modules()`` method. This interface is non-standard, so the module also provides implementations for :class:`importlib.machinery.FileFinder` and :class:`zipimport.zipimporter`. .. versionchanged:: 3.3 Updated to be based directly on :mod:`importlib` rather than relying on the package internal PEP 302 import emulation. .. function:: walk_packages(path=None, prefix='', onerror=None) Yields :class:`ModuleInfo` for all modules recursively on *path*, or, if *path* is ``None``, all accessible modules. *path* should be either ``None`` or a list of paths to look for modules in. *prefix* is a string to output on the front of every module name on output. Note that this function must import all *packages* (*not* all modules!) on the given *path*, in order to access the ``__path__`` attribute to find submodules. *onerror* is a function which gets called with one argument (the name of the package which was being imported) if any exception occurs while trying to import a package. If no *onerror* function is supplied, :exc:`ImportError`\s are caught and ignored, while all other exceptions are propagated, terminating the search. Examples:: # list all modules python can access walk_packages() # list all submodules of ctypes walk_packages(ctypes.__path__, ctypes.__name__ + '.') .. note:: Only works for a :term:`finder` which defines an ``iter_modules()`` method. This interface is non-standard, so the module also provides implementations for :class:`importlib.machinery.FileFinder` and :class:`zipimport.zipimporter`. .. versionchanged:: 3.3 Updated to be based directly on :mod:`importlib` rather than relying on the package internal PEP 302 import emulation. .. function:: get_data(package, resource) Get a resource from a package. This is a wrapper for the :term:`loader` :meth:`get_data <importlib.abc.ResourceLoader.get_data>` API. The *package* argument should be the name of a package, in standard module format (``foo.bar``). The *resource* argument should be in the form of a relative filename, using ``/`` as the path separator. The parent directory name ``..`` is not allowed, and nor is a rooted name (starting with a ``/``). The function returns a binary string that is the contents of the specified resource. For packages located in the filesystem, which have already been imported, this is the rough equivalent of:: d = os.path.dirname(sys.modules[package].__file__) data = open(os.path.join(d, resource), 'rb').read() If the package cannot be located or loaded, or it uses a :term:`loader` which does not support :meth:`get_data <importlib.abc.ResourceLoader.get_data>`, then ``None`` is returned. In particular, the :term:`loader` for :term:`namespace packages <namespace package>` does not support :meth:`get_data <importlib.abc.ResourceLoader.get_data>`. PK����� 3]P �� ��"��library/urllib.robotparser.rst.txtnu�[��������:mod:`urllib.robotparser` --- Parser for robots.txt ==================================================== .. module:: urllib.robotparser :synopsis: Load a robots.txt file and answer questions about fetchability of other URLs. .. sectionauthor:: Skip Montanaro <skip@pobox.com> **Source code:** :source:`Lib/urllib/robotparser.py` .. index:: single: WWW single: World Wide Web single: URL single: robots.txt -------------- This module provides a single class, :class:`RobotFileParser`, which answers questions about whether or not a particular user agent can fetch a URL on the Web site that published the :file:`robots.txt` file. For more details on the structure of :file:`robots.txt` files, see http://www.robotstxt.org/orig.html. .. class:: RobotFileParser(url='') This class provides methods to read, parse and answer questions about the :file:`robots.txt` file at *url*. .. method:: set_url(url) Sets the URL referring to a :file:`robots.txt` file. .. method:: read() Reads the :file:`robots.txt` URL and feeds it to the parser. .. method:: parse(lines) Parses the lines argument. .. method:: can_fetch(useragent, url) Returns ``True`` if the *useragent* is allowed to fetch the *url* according to the rules contained in the parsed :file:`robots.txt` file. .. method:: mtime() Returns the time the ``robots.txt`` file was last fetched. This is useful for long-running web spiders that need to check for new ``robots.txt`` files periodically. .. method:: modified() Sets the time the ``robots.txt`` file was last fetched to the current time. .. method:: crawl_delay(useragent) Returns the value of the ``Crawl-delay`` parameter from ``robots.txt`` for the *useragent* in question. If there is no such parameter or it doesn't apply to the *useragent* specified or the ``robots.txt`` entry for this parameter has invalid syntax, return ``None``. .. versionadded:: 3.6 .. method:: request_rate(useragent) Returns the contents of the ``Request-rate`` parameter from ``robots.txt`` as a :term:`named tuple` ``RequestRate(requests, seconds)``. If there is no such parameter or it doesn't apply to the *useragent* specified or the ``robots.txt`` entry for this parameter has invalid syntax, return ``None``. .. versionadded:: 3.6 The following example demonstrates basic use of the :class:`RobotFileParser` class:: >>> import urllib.robotparser >>> rp = urllib.robotparser.RobotFileParser() >>> rp.set_url("http://www.musi-cal.com/robots.txt") >>> rp.read() >>> rrate = rp.request_rate("*") >>> rrate.requests 3 >>> rrate.seconds 20 >>> rp.crawl_delay("*") 6 >>> rp.can_fetch("*", "http://www.musi-cal.com/cgi-bin/search?city=San+Francisco") False >>> rp.can_fetch("*", "http://www.musi-cal.com/") True PK����� 3]h1GH��GH����library/random.rst.txtnu�[��������:mod:`random` --- Generate pseudo-random numbers ================================================ .. module:: random :synopsis: Generate pseudo-random numbers with various common distributions. **Source code:** :source:`Lib/random.py` -------------- This module implements pseudo-random number generators for various distributions. For integers, there is uniform selection from a range. For sequences, there is uniform selection of a random element, a function to generate a random permutation of a list in-place, and a function for random sampling without replacement. On the real line, there are functions to compute uniform, normal (Gaussian), lognormal, negative exponential, gamma, and beta distributions. For generating distributions of angles, the von Mises distribution is available. Almost all module functions depend on the basic function :func:`.random`, which generates a random float uniformly in the semi-open range [0.0, 1.0). Python uses the Mersenne Twister as the core generator. It produces 53-bit precision floats and has a period of 2\*\*19937-1. The underlying implementation in C is both fast and threadsafe. The Mersenne Twister is one of the most extensively tested random number generators in existence. However, being completely deterministic, it is not suitable for all purposes, and is completely unsuitable for cryptographic purposes. The functions supplied by this module are actually bound methods of a hidden instance of the :class:`random.Random` class. You can instantiate your own instances of :class:`Random` to get generators that don't share state. Class :class:`Random` can also be subclassed if you want to use a different basic generator of your own devising: in that case, override the :meth:`~Random.random`, :meth:`~Random.seed`, :meth:`~Random.getstate`, and :meth:`~Random.setstate` methods. Optionally, a new generator can supply a :meth:`~Random.getrandbits` method --- this allows :meth:`randrange` to produce selections over an arbitrarily large range. The :mod:`random` module also provides the :class:`SystemRandom` class which uses the system function :func:`os.urandom` to generate random numbers from sources provided by the operating system. .. warning:: The pseudo-random generators of this module should not be used for security purposes. For security or cryptographic uses, see the :mod:`secrets` module. .. seealso:: M. Matsumoto and T. Nishimura, "Mersenne Twister: A 623-dimensionally equidistributed uniform pseudorandom number generator", ACM Transactions on Modeling and Computer Simulation Vol. 8, No. 1, January pp.3--30 1998. `Complementary-Multiply-with-Carry recipe <https://code.activestate.com/recipes/576707/>`_ for a compatible alternative random number generator with a long period and comparatively simple update operations. Bookkeeping functions --------------------- .. function:: seed(a=None, version=2) Initialize the random number generator. If *a* is omitted or ``None``, the current system time is used. If randomness sources are provided by the operating system, they are used instead of the system time (see the :func:`os.urandom` function for details on availability). If *a* is an int, it is used directly. With version 2 (the default), a :class:`str`, :class:`bytes`, or :class:`bytearray` object gets converted to an :class:`int` and all of its bits are used. With version 1 (provided for reproducing random sequences from older versions of Python), the algorithm for :class:`str` and :class:`bytes` generates a narrower range of seeds. .. versionchanged:: 3.2 Moved to the version 2 scheme which uses all of the bits in a string seed. .. function:: getstate() Return an object capturing the current internal state of the generator. This object can be passed to :func:`setstate` to restore the state. .. function:: setstate(state) *state* should have been obtained from a previous call to :func:`getstate`, and :func:`setstate` restores the internal state of the generator to what it was at the time :func:`getstate` was called. .. function:: getrandbits(k) Returns a Python integer with *k* random bits. This method is supplied with the MersenneTwister generator and some other generators may also provide it as an optional part of the API. When available, :meth:`getrandbits` enables :meth:`randrange` to handle arbitrarily large ranges. Functions for integers ---------------------- .. function:: randrange(stop) randrange(start, stop[, step]) Return a randomly selected element from ``range(start, stop, step)``. This is equivalent to ``choice(range(start, stop, step))``, but doesn't actually build a range object. The positional argument pattern matches that of :func:`range`. Keyword arguments should not be used because the function may use them in unexpected ways. .. versionchanged:: 3.2 :meth:`randrange` is more sophisticated about producing equally distributed values. Formerly it used a style like ``int(random()*n)`` which could produce slightly uneven distributions. .. function:: randint(a, b) Return a random integer *N* such that ``a <= N <= b``. Alias for ``randrange(a, b+1)``. Functions for sequences ----------------------- .. function:: choice(seq) Return a random element from the non-empty sequence *seq*. If *seq* is empty, raises :exc:`IndexError`. .. function:: choices(population, weights=None, *, cum_weights=None, k=1) Return a *k* sized list of elements chosen from the *population* with replacement. If the *population* is empty, raises :exc:`IndexError`. If a *weights* sequence is specified, selections are made according to the relative weights. Alternatively, if a *cum_weights* sequence is given, the selections are made according to the cumulative weights (perhaps computed using :func:`itertools.accumulate`). For example, the relative weights ``[10, 5, 30, 5]`` are equivalent to the cumulative weights ``[10, 15, 45, 50]``. Internally, the relative weights are converted to cumulative weights before making selections, so supplying the cumulative weights saves work. If neither *weights* nor *cum_weights* are specified, selections are made with equal probability. If a weights sequence is supplied, it must be the same length as the *population* sequence. It is a :exc:`TypeError` to specify both *weights* and *cum_weights*. The *weights* or *cum_weights* can use any numeric type that interoperates with the :class:`float` values returned by :func:`random` (that includes integers, floats, and fractions but excludes decimals). .. versionadded:: 3.6 .. function:: shuffle(x[, random]) Shuffle the sequence *x* in place. The optional argument *random* is a 0-argument function returning a random float in [0.0, 1.0); by default, this is the function :func:`.random`. To shuffle an immutable sequence and return a new shuffled list, use ``sample(x, k=len(x))`` instead. Note that even for small ``len(x)``, the total number of permutations of *x* can quickly grow larger than the period of most random number generators. This implies that most permutations of a long sequence can never be generated. For example, a sequence of length 2080 is the largest that can fit within the period of the Mersenne Twister random number generator. .. function:: sample(population, k) Return a *k* length list of unique elements chosen from the population sequence or set. Used for random sampling without replacement. Returns a new list containing elements from the population while leaving the original population unchanged. The resulting list is in selection order so that all sub-slices will also be valid random samples. This allows raffle winners (the sample) to be partitioned into grand prize and second place winners (the subslices). Members of the population need not be :term:`hashable` or unique. If the population contains repeats, then each occurrence is a possible selection in the sample. To choose a sample from a range of integers, use a :func:`range` object as an argument. This is especially fast and space efficient for sampling from a large population: ``sample(range(10000000), k=60)``. If the sample size is larger than the population size, a :exc:`ValueError` is raised. Real-valued distributions ------------------------- The following functions generate specific real-valued distributions. Function parameters are named after the corresponding variables in the distribution's equation, as used in common mathematical practice; most of these equations can be found in any statistics text. .. function:: random() Return the next random floating point number in the range [0.0, 1.0). .. function:: uniform(a, b) Return a random floating point number *N* such that ``a <= N <= b`` for ``a <= b`` and ``b <= N <= a`` for ``b < a``. The end-point value ``b`` may or may not be included in the range depending on floating-point rounding in the equation ``a + (b-a) * random()``. .. function:: triangular(low, high, mode) Return a random floating point number *N* such that ``low <= N <= high`` and with the specified *mode* between those bounds. The *low* and *high* bounds default to zero and one. The *mode* argument defaults to the midpoint between the bounds, giving a symmetric distribution. .. function:: betavariate(alpha, beta) Beta distribution. Conditions on the parameters are ``alpha > 0`` and ``beta > 0``. Returned values range between 0 and 1. .. function:: expovariate(lambd) Exponential distribution. *lambd* is 1.0 divided by the desired mean. It should be nonzero. (The parameter would be called "lambda", but that is a reserved word in Python.) Returned values range from 0 to positive infinity if *lambd* is positive, and from negative infinity to 0 if *lambd* is negative. .. function:: gammavariate(alpha, beta) Gamma distribution. (*Not* the gamma function!) Conditions on the parameters are ``alpha > 0`` and ``beta > 0``. The probability distribution function is:: x ** (alpha - 1) * math.exp(-x / beta) pdf(x) = -------------------------------------- math.gamma(alpha) * beta ** alpha .. function:: gauss(mu, sigma) Gaussian distribution. *mu* is the mean, and *sigma* is the standard deviation. This is slightly faster than the :func:`normalvariate` function defined below. .. function:: lognormvariate(mu, sigma) Log normal distribution. If you take the natural logarithm of this distribution, you'll get a normal distribution with mean *mu* and standard deviation *sigma*. *mu* can have any value, and *sigma* must be greater than zero. .. function:: normalvariate(mu, sigma) Normal distribution. *mu* is the mean, and *sigma* is the standard deviation. .. function:: vonmisesvariate(mu, kappa) *mu* is the mean angle, expressed in radians between 0 and 2\*\ *pi*, and *kappa* is the concentration parameter, which must be greater than or equal to zero. If *kappa* is equal to zero, this distribution reduces to a uniform random angle over the range 0 to 2\*\ *pi*. .. function:: paretovariate(alpha) Pareto distribution. *alpha* is the shape parameter. .. function:: weibullvariate(alpha, beta) Weibull distribution. *alpha* is the scale parameter and *beta* is the shape parameter. Alternative Generator --------------------- .. class:: SystemRandom([seed]) Class that uses the :func:`os.urandom` function for generating random numbers from sources provided by the operating system. Not available on all systems. Does not rely on software state, and sequences are not reproducible. Accordingly, the :meth:`seed` method has no effect and is ignored. The :meth:`getstate` and :meth:`setstate` methods raise :exc:`NotImplementedError` if called. Notes on Reproducibility ------------------------ Sometimes it is useful to be able to reproduce the sequences given by a pseudo random number generator. By re-using a seed value, the same sequence should be reproducible from run to run as long as multiple threads are not running. Most of the random module's algorithms and seeding functions are subject to change across Python versions, but two aspects are guaranteed not to change: * If a new seeding method is added, then a backward compatible seeder will be offered. * The generator's :meth:`~Random.random` method will continue to produce the same sequence when the compatible seeder is given the same seed. .. _random-examples: Examples and Recipes -------------------- Basic examples:: >>> random() # Random float: 0.0 <= x < 1.0 0.37444887175646646 >>> uniform(2.5, 10.0) # Random float: 2.5 <= x < 10.0 3.1800146073117523 >>> expovariate(1 / 5) # Interval between arrivals averaging 5 seconds 5.148957571865031 >>> randrange(10) # Integer from 0 to 9 inclusive 7 >>> randrange(0, 101, 2) # Even integer from 0 to 100 inclusive 26 >>> choice(['win', 'lose', 'draw']) # Single random element from a sequence 'draw' >>> deck = 'ace two three four'.split() >>> shuffle(deck) # Shuffle a list >>> deck ['four', 'two', 'ace', 'three'] >>> sample([10, 20, 30, 40, 50], k=4) # Four samples without replacement [40, 10, 50, 30] Simulations:: >>> # Six roulette wheel spins (weighted sampling with replacement) >>> choices(['red', 'black', 'green'], [18, 18, 2], k=6) ['red', 'green', 'black', 'black', 'red', 'black'] >>> # Deal 20 cards without replacement from a deck of 52 playing cards >>> # and determine the proportion of cards with a ten-value >>> # (a ten, jack, queen, or king). >>> deck = collections.Counter(tens=16, low_cards=36) >>> seen = sample(list(deck.elements()), k=20) >>> seen.count('tens') / 20 0.15 >>> # Estimate the probability of getting 5 or more heads from 7 spins >>> # of a biased coin that settles on heads 60% of the time. >>> trial = lambda: choices('HT', cum_weights=(0.60, 1.00), k=7).count('H') >= 5 >>> sum(trial() for i in range(10000)) / 10000 0.4169 >>> # Probability of the median of 5 samples being in middle two quartiles >>> trial = lambda : 2500 <= sorted(choices(range(10000), k=5))[2] < 7500 >>> sum(trial() for i in range(10000)) / 10000 0.7958 Example of `statistical bootstrapping <https://en.wikipedia.org/wiki/Bootstrapping_(statistics)>`_ using resampling with replacement to estimate a confidence interval for the mean of a sample of size five:: # http://statistics.about.com/od/Applications/a/Example-Of-Bootstrapping.htm from statistics import mean from random import choices data = 1, 2, 4, 4, 10 means = sorted(mean(choices(data, k=5)) for i in range(20)) print(f'The sample mean of {mean(data):.1f} has a 90% confidence ' f'interval from {means[1]:.1f} to {means[-2]:.1f}') Example of a `resampling permutation test <https://en.wikipedia.org/wiki/Resampling_(statistics)#Permutation_tests>`_ to determine the statistical significance or `p-value <https://en.wikipedia.org/wiki/P-value>`_ of an observed difference between the effects of a drug versus a placebo:: # Example from "Statistics is Easy" by Dennis Shasha and Manda Wilson from statistics import mean from random import shuffle drug = [54, 73, 53, 70, 73, 68, 52, 65, 65] placebo = [54, 51, 58, 44, 55, 52, 42, 47, 58, 46] observed_diff = mean(drug) - mean(placebo) n = 10000 count = 0 combined = drug + placebo for i in range(n): shuffle(combined) new_diff = mean(combined[:len(drug)]) - mean(combined[len(drug):]) count += (new_diff >= observed_diff) print(f'{n} label reshufflings produced only {count} instances with a difference') print(f'at least as extreme as the observed difference of {observed_diff:.1f}.') print(f'The one-sided p-value of {count / n:.4f} leads us to reject the null') print(f'hypothesis that there is no difference between the drug and the placebo.') Simulation of arrival times and service deliveries in a single server queue:: from random import expovariate, gauss from statistics import mean, median, stdev average_arrival_interval = 5.6 average_service_time = 5.0 stdev_service_time = 0.5 num_waiting = 0 arrivals = [] starts = [] arrival = service_end = 0.0 for i in range(20000): if arrival <= service_end: num_waiting += 1 arrival += expovariate(1.0 / average_arrival_interval) arrivals.append(arrival) else: num_waiting -= 1 service_start = service_end if num_waiting else arrival service_time = gauss(average_service_time, stdev_service_time) service_end = service_start + service_time starts.append(service_start) waits = [start - arrival for arrival, start in zip(arrivals, starts)] print(f'Mean wait: {mean(waits):.1f}. Stdev wait: {stdev(waits):.1f}.') print(f'Median wait: {median(waits):.1f}. Max wait: {max(waits):.1f}.') .. seealso:: `Statistics for Hackers <https://www.youtube.com/watch?v=Iq9DzN6mvYA>`_ a video tutorial by `Jake Vanderplas <https://us.pycon.org/2016/speaker/profile/295/>`_ on statistical analysis using just a few fundamental concepts including simulation, sampling, shuffling, and cross-validation. `Economics Simulation <http://nbviewer.jupyter.org/url/norvig.com/ipython/Economics.ipynb>`_ a simulation of a marketplace by `Peter Norvig <http://norvig.com/bio.html>`_ that shows effective use of many of the tools and distributions provided by this module (gauss, uniform, sample, betavariate, choice, triangular, and randrange). `A Concrete Introduction to Probability (using Python) <http://nbviewer.jupyter.org/url/norvig.com/ipython/Probability.ipynb>`_ a tutorial by `Peter Norvig <http://norvig.com/bio.html>`_ covering the basics of probability theory, how to write simulations, and how to perform data analysis using Python. PK����� 3]7{��{����library/ensurepip.rst.txtnu�[��������:mod:`ensurepip` --- Bootstrapping the ``pip`` installer ======================================================== .. module:: ensurepip :synopsis: Bootstrapping the "pip" installer into an existing Python installation or virtual environment. .. versionadded:: 3.4 -------------- The :mod:`ensurepip` package provides support for bootstrapping the ``pip`` installer into an existing Python installation or virtual environment. This bootstrapping approach reflects the fact that ``pip`` is an independent project with its own release cycle, and the latest available stable version is bundled with maintenance and feature releases of the CPython reference interpreter. In most cases, end users of Python shouldn't need to invoke this module directly (as ``pip`` should be bootstrapped by default), but it may be needed if installing ``pip`` was skipped when installing Python (or when creating a virtual environment) or after explicitly uninstalling ``pip``. .. note:: This module *does not* access the internet. All of the components needed to bootstrap ``pip`` are included as internal parts of the package. .. seealso:: :ref:`installing-index` The end user guide for installing Python packages :pep:`453`: Explicit bootstrapping of pip in Python installations The original rationale and specification for this module. Command line interface ---------------------- The command line interface is invoked using the interpreter's ``-m`` switch. The simplest possible invocation is:: python -m ensurepip This invocation will install ``pip`` if it is not already installed, but otherwise does nothing. To ensure the installed version of ``pip`` is at least as recent as the one bundled with ``ensurepip``, pass the ``--upgrade`` option:: python -m ensurepip --upgrade By default, ``pip`` is installed into the current virtual environment (if one is active) or into the system site packages (if there is no active virtual environment). The installation location can be controlled through two additional command line options: * ``--root <dir>``: Installs ``pip`` relative to the given root directory rather than the root of the currently active virtual environment (if any) or the default root for the current Python installation. * ``--user``: Installs ``pip`` into the user site packages directory rather than globally for the current Python installation (this option is not permitted inside an active virtual environment). By default, the scripts ``pipX`` and ``pipX.Y`` will be installed (where X.Y stands for the version of Python used to invoke ``ensurepip``). The scripts installed can be controlled through two additional command line options: * ``--altinstall``: if an alternate installation is requested, the ``pipX`` script will *not* be installed. * ``--default-pip``: if a "default pip" installation is requested, the ``pip`` script will be installed in addition to the two regular scripts. Providing both of the script selection options will trigger an exception. .. versionchanged:: 3.6.3 The exit status is non-zero if the command fails. Module API ---------- :mod:`ensurepip` exposes two functions for programmatic use: .. function:: version() Returns a string specifying the bundled version of pip that will be installed when bootstrapping an environment. .. function:: bootstrap(root=None, upgrade=False, user=False, \ altinstall=False, default_pip=False, \ verbosity=0) Bootstraps ``pip`` into the current or designated environment. *root* specifies an alternative root directory to install relative to. If *root* is ``None``, then installation uses the default install location for the current environment. *upgrade* indicates whether or not to upgrade an existing installation of an earlier version of ``pip`` to the bundled version. *user* indicates whether to use the user scheme rather than installing globally. By default, the scripts ``pipX`` and ``pipX.Y`` will be installed (where X.Y stands for the current version of Python). If *altinstall* is set, then ``pipX`` will *not* be installed. If *default_pip* is set, then ``pip`` will be installed in addition to the two regular scripts. Setting both *altinstall* and *default_pip* will trigger :exc:`ValueError`. *verbosity* controls the level of output to :data:`sys.stdout` from the bootstrapping operation. .. note:: The bootstrapping process has side effects on both ``sys.path`` and ``os.environ``. Invoking the command line interface in a subprocess instead allows these side effects to be avoided. .. note:: The bootstrapping process may install additional modules required by ``pip``, but other software should not assume those dependencies will always be present by default (as the dependencies may be removed in a future version of ``pip``). PK����� 3]%������library/telnetlib.rst.txtnu�[��������:mod:`telnetlib` --- Telnet client ================================== .. module:: telnetlib :synopsis: Telnet client class. .. sectionauthor:: Skip Montanaro <skip@pobox.com> **Source code:** :source:`Lib/telnetlib.py` .. index:: single: protocol; Telnet -------------- The :mod:`telnetlib` module provides a :class:`Telnet` class that implements the Telnet protocol. See :rfc:`854` for details about the protocol. In addition, it provides symbolic constants for the protocol characters (see below), and for the telnet options. The symbolic names of the telnet options follow the definitions in ``arpa/telnet.h``, with the leading ``TELOPT_`` removed. For symbolic names of options which are traditionally not included in ``arpa/telnet.h``, see the module source itself. The symbolic constants for the telnet commands are: IAC, DONT, DO, WONT, WILL, SE (Subnegotiation End), NOP (No Operation), DM (Data Mark), BRK (Break), IP (Interrupt process), AO (Abort output), AYT (Are You There), EC (Erase Character), EL (Erase Line), GA (Go Ahead), SB (Subnegotiation Begin). .. class:: Telnet(host=None, port=0[, timeout]) :class:`Telnet` represents a connection to a Telnet server. The instance is initially not connected by default; the :meth:`open` method must be used to establish a connection. Alternatively, the host name and optional port number can be passed to the constructor too, in which case the connection to the server will be established before the constructor returns. The optional *timeout* parameter specifies a timeout in seconds for blocking operations like the connection attempt (if not specified, the global default timeout setting will be used). Do not reopen an already connected instance. This class has many :meth:`read_\*` methods. Note that some of them raise :exc:`EOFError` when the end of the connection is read, because they can return an empty string for other reasons. See the individual descriptions below. A :class:`Telnet` object is a context manager and can be used in a :keyword:`with` statement. When the :keyword:`with` block ends, the :meth:`close` method is called:: >>> from telnetlib import Telnet >>> with Telnet('localhost', 23) as tn: ... tn.interact() ... .. versionchanged:: 3.6 Context manager support added .. seealso:: :rfc:`854` - Telnet Protocol Specification Definition of the Telnet protocol. .. _telnet-objects: Telnet Objects -------------- :class:`Telnet` instances have the following methods: .. method:: Telnet.read_until(expected, timeout=None) Read until a given byte string, *expected*, is encountered or until *timeout* seconds have passed. When no match is found, return whatever is available instead, possibly empty bytes. Raise :exc:`EOFError` if the connection is closed and no cooked data is available. .. method:: Telnet.read_all() Read all data until EOF as bytes; block until connection closed. .. method:: Telnet.read_some() Read at least one byte of cooked data unless EOF is hit. Return ``b''`` if EOF is hit. Block if no data is immediately available. .. method:: Telnet.read_very_eager() Read everything that can be without blocking in I/O (eager). Raise :exc:`EOFError` if connection closed and no cooked data available. Return ``b''`` if no cooked data available otherwise. Do not block unless in the midst of an IAC sequence. .. method:: Telnet.read_eager() Read readily available data. Raise :exc:`EOFError` if connection closed and no cooked data available. Return ``b''`` if no cooked data available otherwise. Do not block unless in the midst of an IAC sequence. .. method:: Telnet.read_lazy() Process and return data already in the queues (lazy). Raise :exc:`EOFError` if connection closed and no data available. Return ``b''`` if no cooked data available otherwise. Do not block unless in the midst of an IAC sequence. .. method:: Telnet.read_very_lazy() Return any data available in the cooked queue (very lazy). Raise :exc:`EOFError` if connection closed and no data available. Return ``b''`` if no cooked data available otherwise. This method never blocks. .. method:: Telnet.read_sb_data() Return the data collected between a SB/SE pair (suboption begin/end). The callback should access these data when it was invoked with a ``SE`` command. This method never blocks. .. method:: Telnet.open(host, port=0[, timeout]) Connect to a host. The optional second argument is the port number, which defaults to the standard Telnet port (23). The optional *timeout* parameter specifies a timeout in seconds for blocking operations like the connection attempt (if not specified, the global default timeout setting will be used). Do not try to reopen an already connected instance. .. method:: Telnet.msg(msg, *args) Print a debug message when the debug level is ``>`` 0. If extra arguments are present, they are substituted in the message using the standard string formatting operator. .. method:: Telnet.set_debuglevel(debuglevel) Set the debug level. The higher the value of *debuglevel*, the more debug output you get (on ``sys.stdout``). .. method:: Telnet.close() Close the connection. .. method:: Telnet.get_socket() Return the socket object used internally. .. method:: Telnet.fileno() Return the file descriptor of the socket object used internally. .. method:: Telnet.write(buffer) Write a byte string to the socket, doubling any IAC characters. This can block if the connection is blocked. May raise :exc:`OSError` if the connection is closed. .. versionchanged:: 3.3 This method used to raise :exc:`socket.error`, which is now an alias of :exc:`OSError`. .. method:: Telnet.interact() Interaction function, emulates a very dumb Telnet client. .. method:: Telnet.mt_interact() Multithreaded version of :meth:`interact`. .. method:: Telnet.expect(list, timeout=None) Read until one from a list of a regular expressions matches. The first argument is a list of regular expressions, either compiled (:ref:`regex objects <re-objects>`) or uncompiled (byte strings). The optional second argument is a timeout, in seconds; the default is to block indefinitely. Return a tuple of three items: the index in the list of the first regular expression that matches; the match object returned; and the bytes read up till and including the match. If end of file is found and no bytes were read, raise :exc:`EOFError`. Otherwise, when nothing matches, return ``(-1, None, data)`` where *data* is the bytes received so far (may be empty bytes if a timeout happened). If a regular expression ends with a greedy match (such as ``.*``) or if more than one expression can match the same input, the results are non-deterministic, and may depend on the I/O timing. .. method:: Telnet.set_option_negotiation_callback(callback) Each time a telnet option is read on the input flow, this *callback* (if set) is called with the following parameters: callback(telnet socket, command (DO/DONT/WILL/WONT), option). No other action is done afterwards by telnetlib. .. _telnet-example: Telnet Example -------------- .. sectionauthor:: Peter Funk <pf@artcom-gmbh.de> A simple example illustrating typical use:: import getpass import telnetlib HOST = "localhost" user = input("Enter your remote account: ") password = getpass.getpass() tn = telnetlib.Telnet(HOST) tn.read_until(b"login: ") tn.write(user.encode('ascii') + b"\n") if password: tn.read_until(b"Password: ") tn.write(password.encode('ascii') + b"\n") tn.write(b"ls\n") tn.write(b"exit\n") print(tn.read_all().decode('ascii')) PK����� 3]S������library/crypto.rst.txtnu�[��������.. _crypto: ********************** Cryptographic Services ********************** .. index:: single: cryptography The modules described in this chapter implement various algorithms of a cryptographic nature. They are available at the discretion of the installation. On Unix systems, the :mod:`crypt` module may also be available. Here's an overview: .. toctree:: hashlib.rst hmac.rst secrets.rst PK����� 3]2˲������library/_dummy_thread.rst.txtnu�[��������:mod:`_dummy_thread` --- Drop-in replacement for the :mod:`_thread` module ========================================================================== .. module:: _dummy_thread :synopsis: Drop-in replacement for the _thread module. **Source code:** :source:`Lib/_dummy_thread.py` -------------- This module provides a duplicate interface to the :mod:`_thread` module. It is meant to be imported when the :mod:`_thread` module is not provided on a platform. Suggested usage is:: try: import _thread except ImportError: import _dummy_thread as _thread Be careful to not use this module where deadlock might occur from a thread being created that blocks waiting for another thread to be created. This often occurs with blocking I/O. PK����� 3]w �� ����library/language.rst.txtnu�[��������.. _language: ************************ Python Language Services ************************ Python provides a number of modules to assist in working with the Python language. These modules support tokenizing, parsing, syntax analysis, bytecode disassembly, and various other facilities. These modules include: .. toctree:: parser.rst ast.rst symtable.rst symbol.rst token.rst keyword.rst tokenize.rst tabnanny.rst pyclbr.rst py_compile.rst compileall.rst dis.rst pickletools.rst PK����� 3]S ��S ����library/uu.rst.txtnu�[��������:mod:`uu` --- Encode and decode uuencode files ============================================== .. module:: uu :synopsis: Encode and decode files in uuencode format. .. moduleauthor:: Lance Ellinghouse **Source code:** :source:`Lib/uu.py` -------------- This module encodes and decodes files in uuencode format, allowing arbitrary binary data to be transferred over ASCII-only connections. Wherever a file argument is expected, the methods accept a file-like object. For backwards compatibility, a string containing a pathname is also accepted, and the corresponding file will be opened for reading and writing; the pathname ``'-'`` is understood to mean the standard input or output. However, this interface is deprecated; it's better for the caller to open the file itself, and be sure that, when required, the mode is ``'rb'`` or ``'wb'`` on Windows. .. index:: single: Jansen, Jack single: Ellinghouse, Lance This code was contributed by Lance Ellinghouse, and modified by Jack Jansen. The :mod:`uu` module defines the following functions: .. function:: encode(in_file, out_file, name=None, mode=None) Uuencode file *in_file* into file *out_file*. The uuencoded file will have the header specifying *name* and *mode* as the defaults for the results of decoding the file. The default defaults are taken from *in_file*, or ``'-'`` and ``0o666`` respectively. .. function:: decode(in_file, out_file=None, mode=None, quiet=False) This call decodes uuencoded file *in_file* placing the result on file *out_file*. If *out_file* is a pathname, *mode* is used to set the permission bits if the file must be created. Defaults for *out_file* and *mode* are taken from the uuencode header. However, if the file specified in the header already exists, a :exc:`uu.Error` is raised. :func:`decode` may print a warning to standard error if the input was produced by an incorrect uuencoder and Python could recover from that error. Setting *quiet* to a true value silences this warning. .. exception:: Error() Subclass of :exc:`Exception`, this can be raised by :func:`uu.decode` under various situations, such as described above, but also including a badly formatted header, or truncated input file. .. seealso:: Module :mod:`binascii` Support module containing ASCII-to-binary and binary-to-ASCII conversions. PK����� 3]#^qj��j����library/urllib.parse.rst.txtnu�[��������:mod:`urllib.parse` --- Parse URLs into components ================================================== .. module:: urllib.parse :synopsis: Parse URLs into or assemble them from components. **Source code:** :source:`Lib/urllib/parse.py` .. index:: single: WWW single: World Wide Web single: URL pair: URL; parsing pair: relative; URL -------------- This module defines a standard interface to break Uniform Resource Locator (URL) strings up in components (addressing scheme, network location, path etc.), to combine the components back into a URL string, and to convert a "relative URL" to an absolute URL given a "base URL." The module has been designed to match the Internet RFC on Relative Uniform Resource Locators. It supports the following URL schemes: ``file``, ``ftp``, ``gopher``, ``hdl``, ``http``, ``https``, ``imap``, ``mailto``, ``mms``, ``news``, ``nntp``, ``prospero``, ``rsync``, ``rtsp``, ``rtspu``, ``sftp``, ``shttp``, ``sip``, ``sips``, ``snews``, ``svn``, ``svn+ssh``, ``telnet``, ``wais``, ``ws``, ``wss``. The :mod:`urllib.parse` module defines functions that fall into two broad categories: URL parsing and URL quoting. These are covered in detail in the following sections. URL Parsing ----------- The URL parsing functions focus on splitting a URL string into its components, or on combining URL components into a URL string. .. function:: urlparse(urlstring, scheme='', allow_fragments=True) Parse a URL into six components, returning a 6-tuple. This corresponds to the general structure of a URL: ``scheme://netloc/path;parameters?query#fragment``. Each tuple item is a string, possibly empty. The components are not broken up in smaller parts (for example, the network location is a single string), and % escapes are not expanded. The delimiters as shown above are not part of the result, except for a leading slash in the *path* component, which is retained if present. For example: >>> from urllib.parse import urlparse >>> o = urlparse('http://www.cwi.nl:80/%7Eguido/Python.html') >>> o # doctest: +NORMALIZE_WHITESPACE ParseResult(scheme='http', netloc='www.cwi.nl:80', path='/%7Eguido/Python.html', params='', query='', fragment='') >>> o.scheme 'http' >>> o.port 80 >>> o.geturl() 'http://www.cwi.nl:80/%7Eguido/Python.html' Following the syntax specifications in :rfc:`1808`, urlparse recognizes a netloc only if it is properly introduced by '//'. Otherwise the input is presumed to be a relative URL and thus to start with a path component. >>> from urllib.parse import urlparse >>> urlparse('//www.cwi.nl:80/%7Eguido/Python.html') ParseResult(scheme='', netloc='www.cwi.nl:80', path='/%7Eguido/Python.html', params='', query='', fragment='') >>> urlparse('www.cwi.nl/%7Eguido/Python.html') ParseResult(scheme='', netloc='', path='www.cwi.nl/%7Eguido/Python.html', params='', query='', fragment='') >>> urlparse('help/Python.html') ParseResult(scheme='', netloc='', path='help/Python.html', params='', query='', fragment='') The *scheme* argument gives the default addressing scheme, to be used only if the URL does not specify one. It should be the same type (text or bytes) as *urlstring*, except that the default value ``''`` is always allowed, and is automatically converted to ``b''`` if appropriate. If the *allow_fragments* argument is false, fragment identifiers are not recognized. Instead, they are parsed as part of the path, parameters or query component, and :attr:`fragment` is set to the empty string in the return value. The return value is actually an instance of a subclass of :class:`tuple`. This class has the following additional read-only convenience attributes: +------------------+-------+--------------------------+----------------------+ | Attribute | Index | Value | Value if not present | +==================+=======+==========================+======================+ | :attr:`scheme` | 0 | URL scheme specifier | *scheme* parameter | +------------------+-------+--------------------------+----------------------+ | :attr:`netloc` | 1 | Network location part | empty string | +------------------+-------+--------------------------+----------------------+ | :attr:`path` | 2 | Hierarchical path | empty string | +------------------+-------+--------------------------+----------------------+ | :attr:`params` | 3 | Parameters for last path | empty string | | | | element | | +------------------+-------+--------------------------+----------------------+ | :attr:`query` | 4 | Query component | empty string | +------------------+-------+--------------------------+----------------------+ | :attr:`fragment` | 5 | Fragment identifier | empty string | +------------------+-------+--------------------------+----------------------+ | :attr:`username` | | User name | :const:`None` | +------------------+-------+--------------------------+----------------------+ | :attr:`password` | | Password | :const:`None` | +------------------+-------+--------------------------+----------------------+ | :attr:`hostname` | | Host name (lower case) | :const:`None` | +------------------+-------+--------------------------+----------------------+ | :attr:`port` | | Port number as integer, | :const:`None` | | | | if present | | +------------------+-------+--------------------------+----------------------+ Reading the :attr:`port` attribute will raise a :exc:`ValueError` if an invalid port is specified in the URL. See section :ref:`urlparse-result-object` for more information on the result object. Unmatched square brackets in the :attr:`netloc` attribute will raise a :exc:`ValueError`. .. versionchanged:: 3.2 Added IPv6 URL parsing capabilities. .. versionchanged:: 3.3 The fragment is now parsed for all URL schemes (unless *allow_fragment* is false), in accordance with :rfc:`3986`. Previously, a whitelist of schemes that support fragments existed. .. versionchanged:: 3.6 Out-of-range port numbers now raise :exc:`ValueError`, instead of returning :const:`None`. .. function:: parse_qs(qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace') Parse a query string given as a string argument (data of type :mimetype:`application/x-www-form-urlencoded`). Data are returned as a dictionary. The dictionary keys are the unique query variable names and the values are lists of values for each name. The optional argument *keep_blank_values* is a flag indicating whether blank values in percent-encoded queries should be treated as blank strings. A true value indicates that blanks should be retained as blank strings. The default false value indicates that blank values are to be ignored and treated as if they were not included. The optional argument *strict_parsing* is a flag indicating what to do with parsing errors. If false (the default), errors are silently ignored. If true, errors raise a :exc:`ValueError` exception. The optional *encoding* and *errors* parameters specify how to decode percent-encoded sequences into Unicode characters, as accepted by the :meth:`bytes.decode` method. Use the :func:`urllib.parse.urlencode` function (with the ``doseq`` parameter set to ``True``) to convert such dictionaries into query strings. .. versionchanged:: 3.2 Add *encoding* and *errors* parameters. .. function:: parse_qsl(qs, keep_blank_values=False, strict_parsing=False, encoding='utf-8', errors='replace') Parse a query string given as a string argument (data of type :mimetype:`application/x-www-form-urlencoded`). Data are returned as a list of name, value pairs. The optional argument *keep_blank_values* is a flag indicating whether blank values in percent-encoded queries should be treated as blank strings. A true value indicates that blanks should be retained as blank strings. The default false value indicates that blank values are to be ignored and treated as if they were not included. The optional argument *strict_parsing* is a flag indicating what to do with parsing errors. If false (the default), errors are silently ignored. If true, errors raise a :exc:`ValueError` exception. The optional *encoding* and *errors* parameters specify how to decode percent-encoded sequences into Unicode characters, as accepted by the :meth:`bytes.decode` method. Use the :func:`urllib.parse.urlencode` function to convert such lists of pairs into query strings. .. versionchanged:: 3.2 Add *encoding* and *errors* parameters. .. function:: urlunparse(parts) Construct a URL from a tuple as returned by ``urlparse()``. The *parts* argument can be any six-item iterable. This may result in a slightly different, but equivalent URL, if the URL that was parsed originally had unnecessary delimiters (for example, a ``?`` with an empty query; the RFC states that these are equivalent). .. function:: urlsplit(urlstring, scheme='', allow_fragments=True) This is similar to :func:`urlparse`, but does not split the params from the URL. This should generally be used instead of :func:`urlparse` if the more recent URL syntax allowing parameters to be applied to each segment of the *path* portion of the URL (see :rfc:`2396`) is wanted. A separate function is needed to separate the path segments and parameters. This function returns a 5-tuple: (addressing scheme, network location, path, query, fragment identifier). The return value is actually an instance of a subclass of :class:`tuple`. This class has the following additional read-only convenience attributes: +------------------+-------+-------------------------+----------------------+ | Attribute | Index | Value | Value if not present | +==================+=======+=========================+======================+ | :attr:`scheme` | 0 | URL scheme specifier | *scheme* parameter | +------------------+-------+-------------------------+----------------------+ | :attr:`netloc` | 1 | Network location part | empty string | +------------------+-------+-------------------------+----------------------+ | :attr:`path` | 2 | Hierarchical path | empty string | +------------------+-------+-------------------------+----------------------+ | :attr:`query` | 3 | Query component | empty string | +------------------+-------+-------------------------+----------------------+ | :attr:`fragment` | 4 | Fragment identifier | empty string | +------------------+-------+-------------------------+----------------------+ | :attr:`username` | | User name | :const:`None` | +------------------+-------+-------------------------+----------------------+ | :attr:`password` | | Password | :const:`None` | +------------------+-------+-------------------------+----------------------+ | :attr:`hostname` | | Host name (lower case) | :const:`None` | +------------------+-------+-------------------------+----------------------+ | :attr:`port` | | Port number as integer, | :const:`None` | | | | if present | | +------------------+-------+-------------------------+----------------------+ Reading the :attr:`port` attribute will raise a :exc:`ValueError` if an invalid port is specified in the URL. See section :ref:`urlparse-result-object` for more information on the result object. Unmatched square brackets in the :attr:`netloc` attribute will raise a :exc:`ValueError`. .. versionchanged:: 3.6 Out-of-range port numbers now raise :exc:`ValueError`, instead of returning :const:`None`. .. function:: urlunsplit(parts) Combine the elements of a tuple as returned by :func:`urlsplit` into a complete URL as a string. The *parts* argument can be any five-item iterable. This may result in a slightly different, but equivalent URL, if the URL that was parsed originally had unnecessary delimiters (for example, a ? with an empty query; the RFC states that these are equivalent). .. function:: urljoin(base, url, allow_fragments=True) Construct a full ("absolute") URL by combining a "base URL" (*base*) with another URL (*url*). Informally, this uses components of the base URL, in particular the addressing scheme, the network location and (part of) the path, to provide missing components in the relative URL. For example: >>> from urllib.parse import urljoin >>> urljoin('http://www.cwi.nl/%7Eguido/Python.html', 'FAQ.html') 'http://www.cwi.nl/%7Eguido/FAQ.html' The *allow_fragments* argument has the same meaning and default as for :func:`urlparse`. .. note:: If *url* is an absolute URL (that is, starting with ``//`` or ``scheme://``), the *url*'s host name and/or scheme will be present in the result. For example: .. doctest:: >>> urljoin('http://www.cwi.nl/%7Eguido/Python.html', ... '//www.python.org/%7Eguido') 'http://www.python.org/%7Eguido' If you do not want that behavior, preprocess the *url* with :func:`urlsplit` and :func:`urlunsplit`, removing possible *scheme* and *netloc* parts. .. versionchanged:: 3.5 Behaviour updated to match the semantics defined in :rfc:`3986`. .. function:: urldefrag(url) If *url* contains a fragment identifier, return a modified version of *url* with no fragment identifier, and the fragment identifier as a separate string. If there is no fragment identifier in *url*, return *url* unmodified and an empty string. The return value is actually an instance of a subclass of :class:`tuple`. This class has the following additional read-only convenience attributes: +------------------+-------+-------------------------+----------------------+ | Attribute | Index | Value | Value if not present | +==================+=======+=========================+======================+ | :attr:`url` | 0 | URL with no fragment | empty string | +------------------+-------+-------------------------+----------------------+ | :attr:`fragment` | 1 | Fragment identifier | empty string | +------------------+-------+-------------------------+----------------------+ See section :ref:`urlparse-result-object` for more information on the result object. .. versionchanged:: 3.2 Result is a structured object rather than a simple 2-tuple. .. _parsing-ascii-encoded-bytes: Parsing ASCII Encoded Bytes --------------------------- The URL parsing functions were originally designed to operate on character strings only. In practice, it is useful to be able to manipulate properly quoted and encoded URLs as sequences of ASCII bytes. Accordingly, the URL parsing functions in this module all operate on :class:`bytes` and :class:`bytearray` objects in addition to :class:`str` objects. If :class:`str` data is passed in, the result will also contain only :class:`str` data. If :class:`bytes` or :class:`bytearray` data is passed in, the result will contain only :class:`bytes` data. Attempting to mix :class:`str` data with :class:`bytes` or :class:`bytearray` in a single function call will result in a :exc:`TypeError` being raised, while attempting to pass in non-ASCII byte values will trigger :exc:`UnicodeDecodeError`. To support easier conversion of result objects between :class:`str` and :class:`bytes`, all return values from URL parsing functions provide either an :meth:`encode` method (when the result contains :class:`str` data) or a :meth:`decode` method (when the result contains :class:`bytes` data). The signatures of these methods match those of the corresponding :class:`str` and :class:`bytes` methods (except that the default encoding is ``'ascii'`` rather than ``'utf-8'``). Each produces a value of a corresponding type that contains either :class:`bytes` data (for :meth:`encode` methods) or :class:`str` data (for :meth:`decode` methods). Applications that need to operate on potentially improperly quoted URLs that may contain non-ASCII data will need to do their own decoding from bytes to characters before invoking the URL parsing methods. The behaviour described in this section applies only to the URL parsing functions. The URL quoting functions use their own rules when producing or consuming byte sequences as detailed in the documentation of the individual URL quoting functions. .. versionchanged:: 3.2 URL parsing functions now accept ASCII encoded byte sequences .. _urlparse-result-object: Structured Parse Results ------------------------ The result objects from the :func:`urlparse`, :func:`urlsplit` and :func:`urldefrag` functions are subclasses of the :class:`tuple` type. These subclasses add the attributes listed in the documentation for those functions, the encoding and decoding support described in the previous section, as well as an additional method: .. method:: urllib.parse.SplitResult.geturl() Return the re-combined version of the original URL as a string. This may differ from the original URL in that the scheme may be normalized to lower case and empty components may be dropped. Specifically, empty parameters, queries, and fragment identifiers will be removed. For :func:`urldefrag` results, only empty fragment identifiers will be removed. For :func:`urlsplit` and :func:`urlparse` results, all noted changes will be made to the URL returned by this method. The result of this method remains unchanged if passed back through the original parsing function: >>> from urllib.parse import urlsplit >>> url = 'HTTP://www.Python.org/doc/#' >>> r1 = urlsplit(url) >>> r1.geturl() 'http://www.Python.org/doc/' >>> r2 = urlsplit(r1.geturl()) >>> r2.geturl() 'http://www.Python.org/doc/' The following classes provide the implementations of the structured parse results when operating on :class:`str` objects: .. class:: DefragResult(url, fragment) Concrete class for :func:`urldefrag` results containing :class:`str` data. The :meth:`encode` method returns a :class:`DefragResultBytes` instance. .. versionadded:: 3.2 .. class:: ParseResult(scheme, netloc, path, params, query, fragment) Concrete class for :func:`urlparse` results containing :class:`str` data. The :meth:`encode` method returns a :class:`ParseResultBytes` instance. .. class:: SplitResult(scheme, netloc, path, query, fragment) Concrete class for :func:`urlsplit` results containing :class:`str` data. The :meth:`encode` method returns a :class:`SplitResultBytes` instance. The following classes provide the implementations of the parse results when operating on :class:`bytes` or :class:`bytearray` objects: .. class:: DefragResultBytes(url, fragment) Concrete class for :func:`urldefrag` results containing :class:`bytes` data. The :meth:`decode` method returns a :class:`DefragResult` instance. .. versionadded:: 3.2 .. class:: ParseResultBytes(scheme, netloc, path, params, query, fragment) Concrete class for :func:`urlparse` results containing :class:`bytes` data. The :meth:`decode` method returns a :class:`ParseResult` instance. .. versionadded:: 3.2 .. class:: SplitResultBytes(scheme, netloc, path, query, fragment) Concrete class for :func:`urlsplit` results containing :class:`bytes` data. The :meth:`decode` method returns a :class:`SplitResult` instance. .. versionadded:: 3.2 URL Quoting ----------- The URL quoting functions focus on taking program data and making it safe for use as URL components by quoting special characters and appropriately encoding non-ASCII text. They also support reversing these operations to recreate the original data from the contents of a URL component if that task isn't already covered by the URL parsing functions above. .. function:: quote(string, safe='/', encoding=None, errors=None) Replace special characters in *string* using the ``%xx`` escape. Letters, digits, and the characters ``'_.-'`` are never quoted. By default, this function is intended for quoting the path section of URL. The optional *safe* parameter specifies additional ASCII characters that should not be quoted --- its default value is ``'/'``. *string* may be either a :class:`str` or a :class:`bytes`. The optional *encoding* and *errors* parameters specify how to deal with non-ASCII characters, as accepted by the :meth:`str.encode` method. *encoding* defaults to ``'utf-8'``. *errors* defaults to ``'strict'``, meaning unsupported characters raise a :class:`UnicodeEncodeError`. *encoding* and *errors* must not be supplied if *string* is a :class:`bytes`, or a :class:`TypeError` is raised. Note that ``quote(string, safe, encoding, errors)`` is equivalent to ``quote_from_bytes(string.encode(encoding, errors), safe)``. Example: ``quote('/El Niño/')`` yields ``'/El%20Ni%C3%B1o/'``. .. function:: quote_plus(string, safe='', encoding=None, errors=None) Like :func:`quote`, but also replace spaces by plus signs, as required for quoting HTML form values when building up a query string to go into a URL. Plus signs in the original string are escaped unless they are included in *safe*. It also does not have *safe* default to ``'/'``. Example: ``quote_plus('/El Niño/')`` yields ``'%2FEl+Ni%C3%B1o%2F'``. .. function:: quote_from_bytes(bytes, safe='/') Like :func:`quote`, but accepts a :class:`bytes` object rather than a :class:`str`, and does not perform string-to-bytes encoding. Example: ``quote_from_bytes(b'a&\xef')`` yields ``'a%26%EF'``. .. function:: unquote(string, encoding='utf-8', errors='replace') Replace ``%xx`` escapes by their single-character equivalent. The optional *encoding* and *errors* parameters specify how to decode percent-encoded sequences into Unicode characters, as accepted by the :meth:`bytes.decode` method. *string* must be a :class:`str`. *encoding* defaults to ``'utf-8'``. *errors* defaults to ``'replace'``, meaning invalid sequences are replaced by a placeholder character. Example: ``unquote('/El%20Ni%C3%B1o/')`` yields ``'/El Niño/'``. .. function:: unquote_plus(string, encoding='utf-8', errors='replace') Like :func:`unquote`, but also replace plus signs by spaces, as required for unquoting HTML form values. *string* must be a :class:`str`. Example: ``unquote_plus('/El+Ni%C3%B1o/')`` yields ``'/El Niño/'``. .. function:: unquote_to_bytes(string) Replace ``%xx`` escapes by their single-octet equivalent, and return a :class:`bytes` object. *string* may be either a :class:`str` or a :class:`bytes`. If it is a :class:`str`, unescaped non-ASCII characters in *string* are encoded into UTF-8 bytes. Example: ``unquote_to_bytes('a%26%EF')`` yields ``b'a&\xef'``. .. function:: urlencode(query, doseq=False, safe='', encoding=None, \ errors=None, quote_via=quote_plus) Convert a mapping object or a sequence of two-element tuples, which may contain :class:`str` or :class:`bytes` objects, to a percent-encoded ASCII text string. If the resultant string is to be used as a *data* for POST operation with the :func:`~urllib.request.urlopen` function, then it should be encoded to bytes, otherwise it would result in a :exc:`TypeError`. The resulting string is a series of ``key=value`` pairs separated by ``'&'`` characters, where both *key* and *value* are quoted using the *quote_via* function. By default, :func:`quote_plus` is used to quote the values, which means spaces are quoted as a ``'+'`` character and '/' characters are encoded as ``%2F``, which follows the standard for GET requests (``application/x-www-form-urlencoded``). An alternate function that can be passed as *quote_via* is :func:`quote`, which will encode spaces as ``%20`` and not encode '/' characters. For maximum control of what is quoted, use ``quote`` and specify a value for *safe*. When a sequence of two-element tuples is used as the *query* argument, the first element of each tuple is a key and the second is a value. The value element in itself can be a sequence and in that case, if the optional parameter *doseq* is evaluates to ``True``, individual ``key=value`` pairs separated by ``'&'`` are generated for each element of the value sequence for the key. The order of parameters in the encoded string will match the order of parameter tuples in the sequence. The *safe*, *encoding*, and *errors* parameters are passed down to *quote_via* (the *encoding* and *errors* parameters are only passed when a query element is a :class:`str`). To reverse this encoding process, :func:`parse_qs` and :func:`parse_qsl` are provided in this module to parse query strings into Python data structures. Refer to :ref:`urllib examples <urllib-examples>` to find out how urlencode method can be used for generating query string for a URL or data for POST. .. versionchanged:: 3.2 Query parameter supports bytes and string objects. .. versionadded:: 3.5 *quote_via* parameter. .. seealso:: :rfc:`3986` - Uniform Resource Identifiers This is the current standard (STD66). Any changes to urllib.parse module should conform to this. Certain deviations could be observed, which are mostly for backward compatibility purposes and for certain de-facto parsing requirements as commonly observed in major browsers. :rfc:`2732` - Format for Literal IPv6 Addresses in URL's. This specifies the parsing requirements of IPv6 URLs. :rfc:`2396` - Uniform Resource Identifiers (URI): Generic Syntax Document describing the generic syntactic requirements for both Uniform Resource Names (URNs) and Uniform Resource Locators (URLs). :rfc:`2368` - The mailto URL scheme. Parsing requirements for mailto URL schemes. :rfc:`1808` - Relative Uniform Resource Locators This Request For Comments includes the rules for joining an absolute and a relative URL, including a fair number of "Abnormal Examples" which govern the treatment of border cases. :rfc:`1738` - Uniform Resource Locators (URL) This specifies the formal syntax and semantics of absolute URLs. PK����� 3]y����$��library/tkinter.scrolledtext.rst.txtnu�[��������:mod:`tkinter.scrolledtext` --- Scrolled Text Widget ==================================================== .. module:: tkinter.scrolledtext :platform: Tk :synopsis: Text widget with a vertical scroll bar. .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> **Source code:** :source:`Lib/tkinter/scrolledtext.py` -------------- The :mod:`tkinter.scrolledtext` module provides a class of the same name which implements a basic text widget which has a vertical scroll bar configured to do the "right thing." Using the :class:`ScrolledText` class is a lot easier than setting up a text widget and scroll bar directly. The constructor is the same as that of the :class:`tkinter.Text` class. The text widget and scrollbar are packed together in a :class:`Frame`, and the methods of the :class:`Grid` and :class:`Pack` geometry managers are acquired from the :class:`Frame` object. This allows the :class:`ScrolledText` widget to be used directly to achieve most normal geometry management behavior. Should more specific control be necessary, the following attributes are available: .. attribute:: ScrolledText.frame The frame which surrounds the text and scroll bar widgets. .. attribute:: ScrolledText.vbar The scroll bar widget. PK����� 3]NIV��IV����library/tracemalloc.rst.txtnu�[��������:mod:`tracemalloc` --- Trace memory allocations =============================================== .. module:: tracemalloc :synopsis: Trace memory allocations. .. versionadded:: 3.4 **Source code:** :source:`Lib/tracemalloc.py` -------------- The tracemalloc module is a debug tool to trace memory blocks allocated by Python. It provides the following information: * Traceback where an object was allocated * Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks * Compute the differences between two snapshots to detect memory leaks To trace most memory blocks allocated by Python, the module should be started as early as possible by setting the :envvar:`PYTHONTRACEMALLOC` environment variable to ``1``, or by using :option:`-X` ``tracemalloc`` command line option. The :func:`tracemalloc.start` function can be called at runtime to start tracing Python memory allocations. By default, a trace of an allocated memory block only stores the most recent frame (1 frame). To store 25 frames at startup: set the :envvar:`PYTHONTRACEMALLOC` environment variable to ``25``, or use the :option:`-X` ``tracemalloc=25`` command line option. Examples -------- Display the top 10 ^^^^^^^^^^^^^^^^^^ Display the 10 files allocating the most memory:: import tracemalloc tracemalloc.start() # ... run your application ... snapshot = tracemalloc.take_snapshot() top_stats = snapshot.statistics('lineno') print("[ Top 10 ]") for stat in top_stats[:10]: print(stat) Example of output of the Python test suite:: [ Top 10 ] <frozen importlib._bootstrap>:716: size=4855 KiB, count=39328, average=126 B <frozen importlib._bootstrap>:284: size=521 KiB, count=3199, average=167 B /usr/lib/python3.4/collections/__init__.py:368: size=244 KiB, count=2315, average=108 B /usr/lib/python3.4/unittest/case.py:381: size=185 KiB, count=779, average=243 B /usr/lib/python3.4/unittest/case.py:402: size=154 KiB, count=378, average=416 B /usr/lib/python3.4/abc.py:133: size=88.7 KiB, count=347, average=262 B <frozen importlib._bootstrap>:1446: size=70.4 KiB, count=911, average=79 B <frozen importlib._bootstrap>:1454: size=52.0 KiB, count=25, average=2131 B <string>:5: size=49.7 KiB, count=148, average=344 B /usr/lib/python3.4/sysconfig.py:411: size=48.0 KiB, count=1, average=48.0 KiB We can see that Python loaded ``4855 KiB`` data (bytecode and constants) from modules and that the :mod:`collections` module allocated ``244 KiB`` to build :class:`~collections.namedtuple` types. See :meth:`Snapshot.statistics` for more options. Compute differences ^^^^^^^^^^^^^^^^^^^ Take two snapshots and display the differences:: import tracemalloc tracemalloc.start() # ... start your application ... snapshot1 = tracemalloc.take_snapshot() # ... call the function leaking memory ... snapshot2 = tracemalloc.take_snapshot() top_stats = snapshot2.compare_to(snapshot1, 'lineno') print("[ Top 10 differences ]") for stat in top_stats[:10]: print(stat) Example of output before/after running some tests of the Python test suite:: [ Top 10 differences ] <frozen importlib._bootstrap>:716: size=8173 KiB (+4428 KiB), count=71332 (+39369), average=117 B /usr/lib/python3.4/linecache.py:127: size=940 KiB (+940 KiB), count=8106 (+8106), average=119 B /usr/lib/python3.4/unittest/case.py:571: size=298 KiB (+298 KiB), count=589 (+589), average=519 B <frozen importlib._bootstrap>:284: size=1005 KiB (+166 KiB), count=7423 (+1526), average=139 B /usr/lib/python3.4/mimetypes.py:217: size=112 KiB (+112 KiB), count=1334 (+1334), average=86 B /usr/lib/python3.4/http/server.py:848: size=96.0 KiB (+96.0 KiB), count=1 (+1), average=96.0 KiB /usr/lib/python3.4/inspect.py:1465: size=83.5 KiB (+83.5 KiB), count=109 (+109), average=784 B /usr/lib/python3.4/unittest/mock.py:491: size=77.7 KiB (+77.7 KiB), count=143 (+143), average=557 B /usr/lib/python3.4/urllib/parse.py:476: size=71.8 KiB (+71.8 KiB), count=969 (+969), average=76 B /usr/lib/python3.4/contextlib.py:38: size=67.2 KiB (+67.2 KiB), count=126 (+126), average=546 B We can see that Python has loaded ``8173 KiB`` of module data (bytecode and constants), and that this is ``4428 KiB`` more than had been loaded before the tests, when the previous snapshot was taken. Similarly, the :mod:`linecache` module has cached ``940 KiB`` of Python source code to format tracebacks, all of it since the previous snapshot. If the system has little free memory, snapshots can be written on disk using the :meth:`Snapshot.dump` method to analyze the snapshot offline. Then use the :meth:`Snapshot.load` method reload the snapshot. Get the traceback of a memory block ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Code to display the traceback of the biggest memory block:: import tracemalloc # Store 25 frames tracemalloc.start(25) # ... run your application ... snapshot = tracemalloc.take_snapshot() top_stats = snapshot.statistics('traceback') # pick the biggest memory block stat = top_stats[0] print("%s memory blocks: %.1f KiB" % (stat.count, stat.size / 1024)) for line in stat.traceback.format(): print(line) Example of output of the Python test suite (traceback limited to 25 frames):: 903 memory blocks: 870.1 KiB File "<frozen importlib._bootstrap>", line 716 File "<frozen importlib._bootstrap>", line 1036 File "<frozen importlib._bootstrap>", line 934 File "<frozen importlib._bootstrap>", line 1068 File "<frozen importlib._bootstrap>", line 619 File "<frozen importlib._bootstrap>", line 1581 File "<frozen importlib._bootstrap>", line 1614 File "/usr/lib/python3.4/doctest.py", line 101 import pdb File "<frozen importlib._bootstrap>", line 284 File "<frozen importlib._bootstrap>", line 938 File "<frozen importlib._bootstrap>", line 1068 File "<frozen importlib._bootstrap>", line 619 File "<frozen importlib._bootstrap>", line 1581 File "<frozen importlib._bootstrap>", line 1614 File "/usr/lib/python3.4/test/support/__init__.py", line 1728 import doctest File "/usr/lib/python3.4/test/test_pickletools.py", line 21 support.run_doctest(pickletools) File "/usr/lib/python3.4/test/regrtest.py", line 1276 test_runner() File "/usr/lib/python3.4/test/regrtest.py", line 976 display_failure=not verbose) File "/usr/lib/python3.4/test/regrtest.py", line 761 match_tests=ns.match_tests) File "/usr/lib/python3.4/test/regrtest.py", line 1563 main() File "/usr/lib/python3.4/test/__main__.py", line 3 regrtest.main_in_temp_cwd() File "/usr/lib/python3.4/runpy.py", line 73 exec(code, run_globals) File "/usr/lib/python3.4/runpy.py", line 160 "__main__", fname, loader, pkg_name) We can see that the most memory was allocated in the :mod:`importlib` module to load data (bytecode and constants) from modules: ``870.1 KiB``. The traceback is where the :mod:`importlib` loaded data most recently: on the ``import pdb`` line of the :mod:`doctest` module. The traceback may change if a new module is loaded. Pretty top ^^^^^^^^^^ Code to display the 10 lines allocating the most memory with a pretty output, ignoring ``<frozen importlib._bootstrap>`` and ``<unknown>`` files:: import linecache import os import tracemalloc def display_top(snapshot, key_type='lineno', limit=10): snapshot = snapshot.filter_traces(( tracemalloc.Filter(False, "<frozen importlib._bootstrap>"), tracemalloc.Filter(False, "<unknown>"), )) top_stats = snapshot.statistics(key_type) print("Top %s lines" % limit) for index, stat in enumerate(top_stats[:limit], 1): frame = stat.traceback[0] # replace "/path/to/module/file.py" with "module/file.py" filename = os.sep.join(frame.filename.split(os.sep)[-2:]) print("#%s: %s:%s: %.1f KiB" % (index, filename, frame.lineno, stat.size / 1024)) line = linecache.getline(frame.filename, frame.lineno).strip() if line: print(' %s' % line) other = top_stats[limit:] if other: size = sum(stat.size for stat in other) print("%s other: %.1f KiB" % (len(other), size / 1024)) total = sum(stat.size for stat in top_stats) print("Total allocated size: %.1f KiB" % (total / 1024)) tracemalloc.start() # ... run your application ... snapshot = tracemalloc.take_snapshot() display_top(snapshot) Example of output of the Python test suite:: Top 10 lines #1: Lib/base64.py:414: 419.8 KiB _b85chars2 = [(a + b) for a in _b85chars for b in _b85chars] #2: Lib/base64.py:306: 419.8 KiB _a85chars2 = [(a + b) for a in _a85chars for b in _a85chars] #3: collections/__init__.py:368: 293.6 KiB exec(class_definition, namespace) #4: Lib/abc.py:133: 115.2 KiB cls = super().__new__(mcls, name, bases, namespace) #5: unittest/case.py:574: 103.1 KiB testMethod() #6: Lib/linecache.py:127: 95.4 KiB lines = fp.readlines() #7: urllib/parse.py:476: 71.8 KiB for a in _hexdig for b in _hexdig} #8: <string>:5: 62.0 KiB #9: Lib/_weakrefset.py:37: 60.0 KiB self.data = set() #10: Lib/base64.py:142: 59.8 KiB _b32tab2 = [a + b for a in _b32tab for b in _b32tab] 6220 other: 3602.8 KiB Total allocated size: 5303.1 KiB See :meth:`Snapshot.statistics` for more options. API --- Functions ^^^^^^^^^ .. function:: clear_traces() Clear traces of memory blocks allocated by Python. See also :func:`stop`. .. function:: get_object_traceback(obj) Get the traceback where the Python object *obj* was allocated. Return a :class:`Traceback` instance, or ``None`` if the :mod:`tracemalloc` module is not tracing memory allocations or did not trace the allocation of the object. See also :func:`gc.get_referrers` and :func:`sys.getsizeof` functions. .. function:: get_traceback_limit() Get the maximum number of frames stored in the traceback of a trace. The :mod:`tracemalloc` module must be tracing memory allocations to get the limit, otherwise an exception is raised. The limit is set by the :func:`start` function. .. function:: get_traced_memory() Get the current size and peak size of memory blocks traced by the :mod:`tracemalloc` module as a tuple: ``(current: int, peak: int)``. .. function:: get_tracemalloc_memory() Get the memory usage in bytes of the :mod:`tracemalloc` module used to store traces of memory blocks. Return an :class:`int`. .. function:: is_tracing() ``True`` if the :mod:`tracemalloc` module is tracing Python memory allocations, ``False`` otherwise. See also :func:`start` and :func:`stop` functions. .. function:: start(nframe: int=1) Start tracing Python memory allocations: install hooks on Python memory allocators. Collected tracebacks of traces will be limited to *nframe* frames. By default, a trace of a memory block only stores the most recent frame: the limit is ``1``. *nframe* must be greater or equal to ``1``. Storing more than ``1`` frame is only useful to compute statistics grouped by ``'traceback'`` or to compute cumulative statistics: see the :meth:`Snapshot.compare_to` and :meth:`Snapshot.statistics` methods. Storing more frames increases the memory and CPU overhead of the :mod:`tracemalloc` module. Use the :func:`get_tracemalloc_memory` function to measure how much memory is used by the :mod:`tracemalloc` module. The :envvar:`PYTHONTRACEMALLOC` environment variable (``PYTHONTRACEMALLOC=NFRAME``) and the :option:`-X` ``tracemalloc=NFRAME`` command line option can be used to start tracing at startup. See also :func:`stop`, :func:`is_tracing` and :func:`get_traceback_limit` functions. .. function:: stop() Stop tracing Python memory allocations: uninstall hooks on Python memory allocators. Also clears all previously collected traces of memory blocks allocated by Python. Call :func:`take_snapshot` function to take a snapshot of traces before clearing them. See also :func:`start`, :func:`is_tracing` and :func:`clear_traces` functions. .. function:: take_snapshot() Take a snapshot of traces of memory blocks allocated by Python. Return a new :class:`Snapshot` instance. The snapshot does not include memory blocks allocated before the :mod:`tracemalloc` module started to trace memory allocations. Tracebacks of traces are limited to :func:`get_traceback_limit` frames. Use the *nframe* parameter of the :func:`start` function to store more frames. The :mod:`tracemalloc` module must be tracing memory allocations to take a snapshot, see the :func:`start` function. See also the :func:`get_object_traceback` function. DomainFilter ^^^^^^^^^^^^ .. class:: DomainFilter(inclusive: bool, domain: int) Filter traces of memory blocks by their address space (domain). .. versionadded:: 3.6 .. attribute:: inclusive If *inclusive* is ``True`` (include), match memory blocks allocated in the address space :attr:`domain`. If *inclusive* is ``False`` (exclude), match memory blocks not allocated in the address space :attr:`domain`. .. attribute:: domain Address space of a memory block (``int``). Read-only property. Filter ^^^^^^ .. class:: Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False, domain: int=None) Filter on traces of memory blocks. See the :func:`fnmatch.fnmatch` function for the syntax of *filename_pattern*. The ``'.pyc'`` file extension is replaced with ``'.py'``. Examples: * ``Filter(True, subprocess.__file__)`` only includes traces of the :mod:`subprocess` module * ``Filter(False, tracemalloc.__file__)`` excludes traces of the :mod:`tracemalloc` module * ``Filter(False, "<unknown>")`` excludes empty tracebacks .. versionchanged:: 3.5 The ``'.pyo'`` file extension is no longer replaced with ``'.py'``. .. versionchanged:: 3.6 Added the :attr:`domain` attribute. .. attribute:: domain Address space of a memory block (``int`` or ``None``). .. attribute:: inclusive If *inclusive* is ``True`` (include), only match memory blocks allocated in a file with a name matching :attr:`filename_pattern` at line number :attr:`lineno`. If *inclusive* is ``False`` (exclude), ignore memory blocks allocated in a file with a name matching :attr:`filename_pattern` at line number :attr:`lineno`. .. attribute:: lineno Line number (``int``) of the filter. If *lineno* is ``None``, the filter matches any line number. .. attribute:: filename_pattern Filename pattern of the filter (``str``). Read-only property. .. attribute:: all_frames If *all_frames* is ``True``, all frames of the traceback are checked. If *all_frames* is ``False``, only the most recent frame is checked. This attribute has no effect if the traceback limit is ``1``. See the :func:`get_traceback_limit` function and :attr:`Snapshot.traceback_limit` attribute. Frame ^^^^^ .. class:: Frame Frame of a traceback. The :class:`Traceback` class is a sequence of :class:`Frame` instances. .. attribute:: filename Filename (``str``). .. attribute:: lineno Line number (``int``). Snapshot ^^^^^^^^ .. class:: Snapshot Snapshot of traces of memory blocks allocated by Python. The :func:`take_snapshot` function creates a snapshot instance. .. method:: compare_to(old_snapshot: Snapshot, key_type: str, cumulative: bool=False) Compute the differences with an old snapshot. Get statistics as a sorted list of :class:`StatisticDiff` instances grouped by *key_type*. See the :meth:`Snapshot.statistics` method for *key_type* and *cumulative* parameters. The result is sorted from the biggest to the smallest by: absolute value of :attr:`StatisticDiff.size_diff`, :attr:`StatisticDiff.size`, absolute value of :attr:`StatisticDiff.count_diff`, :attr:`Statistic.count` and then by :attr:`StatisticDiff.traceback`. .. method:: dump(filename) Write the snapshot into a file. Use :meth:`load` to reload the snapshot. .. method:: filter_traces(filters) Create a new :class:`Snapshot` instance with a filtered :attr:`traces` sequence, *filters* is a list of :class:`DomainFilter` and :class:`Filter` instances. If *filters* is an empty list, return a new :class:`Snapshot` instance with a copy of the traces. All inclusive filters are applied at once, a trace is ignored if no inclusive filters match it. A trace is ignored if at least one exclusive filter matches it. .. versionchanged:: 3.6 :class:`DomainFilter` instances are now also accepted in *filters*. .. classmethod:: load(filename) Load a snapshot from a file. See also :meth:`dump`. .. method:: statistics(key_type: str, cumulative: bool=False) Get statistics as a sorted list of :class:`Statistic` instances grouped by *key_type*: ===================== ======================== key_type description ===================== ======================== ``'filename'`` filename ``'lineno'`` filename and line number ``'traceback'`` traceback ===================== ======================== If *cumulative* is ``True``, cumulate size and count of memory blocks of all frames of the traceback of a trace, not only the most recent frame. The cumulative mode can only be used with *key_type* equals to ``'filename'`` and ``'lineno'``. The result is sorted from the biggest to the smallest by: :attr:`Statistic.size`, :attr:`Statistic.count` and then by :attr:`Statistic.traceback`. .. attribute:: traceback_limit Maximum number of frames stored in the traceback of :attr:`traces`: result of the :func:`get_traceback_limit` when the snapshot was taken. .. attribute:: traces Traces of all memory blocks allocated by Python: sequence of :class:`Trace` instances. The sequence has an undefined order. Use the :meth:`Snapshot.statistics` method to get a sorted list of statistics. Statistic ^^^^^^^^^ .. class:: Statistic Statistic on memory allocations. :func:`Snapshot.statistics` returns a list of :class:`Statistic` instances. See also the :class:`StatisticDiff` class. .. attribute:: count Number of memory blocks (``int``). .. attribute:: size Total size of memory blocks in bytes (``int``). .. attribute:: traceback Traceback where the memory block was allocated, :class:`Traceback` instance. StatisticDiff ^^^^^^^^^^^^^ .. class:: StatisticDiff Statistic difference on memory allocations between an old and a new :class:`Snapshot` instance. :func:`Snapshot.compare_to` returns a list of :class:`StatisticDiff` instances. See also the :class:`Statistic` class. .. attribute:: count Number of memory blocks in the new snapshot (``int``): ``0`` if the memory blocks have been released in the new snapshot. .. attribute:: count_diff Difference of number of memory blocks between the old and the new snapshots (``int``): ``0`` if the memory blocks have been allocated in the new snapshot. .. attribute:: size Total size of memory blocks in bytes in the new snapshot (``int``): ``0`` if the memory blocks have been released in the new snapshot. .. attribute:: size_diff Difference of total size of memory blocks in bytes between the old and the new snapshots (``int``): ``0`` if the memory blocks have been allocated in the new snapshot. .. attribute:: traceback Traceback where the memory blocks were allocated, :class:`Traceback` instance. Trace ^^^^^ .. class:: Trace Trace of a memory block. The :attr:`Snapshot.traces` attribute is a sequence of :class:`Trace` instances. .. attribute:: size Size of the memory block in bytes (``int``). .. attribute:: traceback Traceback where the memory block was allocated, :class:`Traceback` instance. Traceback ^^^^^^^^^ .. class:: Traceback Sequence of :class:`Frame` instances sorted from the most recent frame to the oldest frame. A traceback contains at least ``1`` frame. If the ``tracemalloc`` module failed to get a frame, the filename ``"<unknown>"`` at line number ``0`` is used. When a snapshot is taken, tracebacks of traces are limited to :func:`get_traceback_limit` frames. See the :func:`take_snapshot` function. The :attr:`Trace.traceback` attribute is an instance of :class:`Traceback` instance. .. method:: format(limit=None) Format the traceback as a list of lines with newlines. Use the :mod:`linecache` module to retrieve lines from the source code. If *limit* is set, only format the *limit* most recent frames. Similar to the :func:`traceback.format_tb` function, except that :meth:`.format` does not include newlines. Example:: print("Traceback (most recent call first):") for line in traceback: print(line) Output:: Traceback (most recent call first): File "test.py", line 9 obj = Object() File "test.py", line 12 tb = tracemalloc.get_object_traceback(f()) PK����� 3]7cZ��Z����library/xmlrpc.client.rst.txtnu�[��������:mod:`xmlrpc.client` --- XML-RPC client access ============================================== .. module:: xmlrpc.client :synopsis: XML-RPC client access. .. moduleauthor:: Fredrik Lundh <fredrik@pythonware.com> .. sectionauthor:: Eric S. Raymond <esr@snark.thyrsus.com> **Source code:** :source:`Lib/xmlrpc/client.py` .. XXX Not everything is documented yet. It might be good to describe Marshaller, Unmarshaller, getparser and Transport. -------------- XML-RPC is a Remote Procedure Call method that uses XML passed via HTTP(S) as a transport. With it, a client can call methods with parameters on a remote server (the server is named by a URI) and get back structured data. This module supports writing XML-RPC client code; it handles all the details of translating between conformable Python objects and XML on the wire. .. warning:: The :mod:`xmlrpc.client` module is not secure against maliciously constructed data. If you need to parse untrusted or unauthenticated data see :ref:`xml-vulnerabilities`. .. versionchanged:: 3.5 For HTTPS URIs, :mod:`xmlrpc.client` now performs all the necessary certificate and hostname checks by default. .. class:: ServerProxy(uri, transport=None, encoding=None, verbose=False, \ allow_none=False, use_datetime=False, \ use_builtin_types=False, *, context=None) .. versionchanged:: 3.3 The *use_builtin_types* flag was added. A :class:`ServerProxy` instance is an object that manages communication with a remote XML-RPC server. The required first argument is a URI (Uniform Resource Indicator), and will normally be the URL of the server. The optional second argument is a transport factory instance; by default it is an internal :class:`SafeTransport` instance for https: URLs and an internal HTTP :class:`Transport` instance otherwise. The optional third argument is an encoding, by default UTF-8. The optional fourth argument is a debugging flag. The following parameters govern the use of the returned proxy instance. If *allow_none* is true, the Python constant ``None`` will be translated into XML; the default behaviour is for ``None`` to raise a :exc:`TypeError`. This is a commonly-used extension to the XML-RPC specification, but isn't supported by all clients and servers; see `http://ontosys.com/xml-rpc/extensions.php <https://web.archive.org/web/20130120074804/http://ontosys.com/xml-rpc/extensions.php>`_ for a description. The *use_builtin_types* flag can be used to cause date/time values to be presented as :class:`datetime.datetime` objects and binary data to be presented as :class:`bytes` objects; this flag is false by default. :class:`datetime.datetime`, :class:`bytes` and :class:`bytearray` objects may be passed to calls. The obsolete *use_datetime* flag is similar to *use_builtin_types* but it applies only to date/time values. Both the HTTP and HTTPS transports support the URL syntax extension for HTTP Basic Authentication: ``http://user:pass@host:port/path``. The ``user:pass`` portion will be base64-encoded as an HTTP 'Authorization' header, and sent to the remote server as part of the connection process when invoking an XML-RPC method. You only need to use this if the remote server requires a Basic Authentication user and password. If an HTTPS URL is provided, *context* may be :class:`ssl.SSLContext` and configures the SSL settings of the underlying HTTPS connection. The returned instance is a proxy object with methods that can be used to invoke corresponding RPC calls on the remote server. If the remote server supports the introspection API, the proxy can also be used to query the remote server for the methods it supports (service discovery) and fetch other server-associated metadata. Types that are conformable (e.g. that can be marshalled through XML), include the following (and except where noted, they are unmarshalled as the same Python type): .. tabularcolumns:: |l|L| +----------------------+-------------------------------------------------------+ | XML-RPC type | Python type | +======================+=======================================================+ | ``boolean`` | :class:`bool` | +----------------------+-------------------------------------------------------+ | ``int``, ``i1``, | :class:`int` in range from -2147483648 to 2147483647. | | ``i2``, ``i4``, | Values get the ``<int>`` tag. | | ``i8`` or | | | ``biginteger`` | | +----------------------+-------------------------------------------------------+ | ``double`` or | :class:`float`. Values get the ``<double>`` tag. | | ``float`` | | +----------------------+-------------------------------------------------------+ | ``string`` | :class:`str` | +----------------------+-------------------------------------------------------+ | ``array`` | :class:`list` or :class:`tuple` containing | | | conformable elements. Arrays are returned as | | | :class:`lists <list>`. | +----------------------+-------------------------------------------------------+ | ``struct`` | :class:`dict`. Keys must be strings, values may be | | | any conformable type. Objects of user-defined | | | classes can be passed in; only their | | | :attr:`~object.__dict__` attribute is transmitted. | +----------------------+-------------------------------------------------------+ | ``dateTime.iso8601`` | :class:`DateTime` or :class:`datetime.datetime`. | | | Returned type depends on values of | | | *use_builtin_types* and *use_datetime* flags. | +----------------------+-------------------------------------------------------+ | ``base64`` | :class:`Binary`, :class:`bytes` or | | | :class:`bytearray`. Returned type depends on the | | | value of the *use_builtin_types* flag. | +----------------------+-------------------------------------------------------+ | ``nil`` | The ``None`` constant. Passing is allowed only if | | | *allow_none* is true. | +----------------------+-------------------------------------------------------+ | ``bigdecimal`` | :class:`decimal.Decimal`. Returned type only. | +----------------------+-------------------------------------------------------+ This is the full set of data types supported by XML-RPC. Method calls may also raise a special :exc:`Fault` instance, used to signal XML-RPC server errors, or :exc:`ProtocolError` used to signal an error in the HTTP/HTTPS transport layer. Both :exc:`Fault` and :exc:`ProtocolError` derive from a base class called :exc:`Error`. Note that the xmlrpc client module currently does not marshal instances of subclasses of built-in types. When passing strings, characters special to XML such as ``<``, ``>``, and ``&`` will be automatically escaped. However, it's the caller's responsibility to ensure that the string is free of characters that aren't allowed in XML, such as the control characters with ASCII values between 0 and 31 (except, of course, tab, newline and carriage return); failing to do this will result in an XML-RPC request that isn't well-formed XML. If you have to pass arbitrary bytes via XML-RPC, use :class:`bytes` or :class:`bytearray` classes or the :class:`Binary` wrapper class described below. :class:`Server` is retained as an alias for :class:`ServerProxy` for backwards compatibility. New code should use :class:`ServerProxy`. .. versionchanged:: 3.5 Added the *context* argument. .. versionchanged:: 3.6 Added support of type tags with prefixes (e.g. ``ex:nil``). Added support of unmarshalling additional types used by Apache XML-RPC implementation for numerics: ``i1``, ``i2``, ``i8``, ``biginteger``, ``float`` and ``bigdecimal``. See http://ws.apache.org/xmlrpc/types.html for a description. .. seealso:: `XML-RPC HOWTO <http://www.tldp.org/HOWTO/XML-RPC-HOWTO/index.html>`_ A good description of XML-RPC operation and client software in several languages. Contains pretty much everything an XML-RPC client developer needs to know. `XML-RPC Introspection <http://xmlrpc-c.sourceforge.net/introspection.html>`_ Describes the XML-RPC protocol extension for introspection. `XML-RPC Specification <http://xmlrpc.scripting.com/spec.html>`_ The official specification. `Unofficial XML-RPC Errata <http://effbot.org/zone/xmlrpc-errata.htm>`_ Fredrik Lundh's "unofficial errata, intended to clarify certain details in the XML-RPC specification, as well as hint at 'best practices' to use when designing your own XML-RPC implementations." .. _serverproxy-objects: ServerProxy Objects ------------------- A :class:`ServerProxy` instance has a method corresponding to each remote procedure call accepted by the XML-RPC server. Calling the method performs an RPC, dispatched by both name and argument signature (e.g. the same method name can be overloaded with multiple argument signatures). The RPC finishes by returning a value, which may be either returned data in a conformant type or a :class:`Fault` or :class:`ProtocolError` object indicating an error. Servers that support the XML introspection API support some common methods grouped under the reserved :attr:`~ServerProxy.system` attribute: .. method:: ServerProxy.system.listMethods() This method returns a list of strings, one for each (non-system) method supported by the XML-RPC server. .. method:: ServerProxy.system.methodSignature(name) This method takes one parameter, the name of a method implemented by the XML-RPC server. It returns an array of possible signatures for this method. A signature is an array of types. The first of these types is the return type of the method, the rest are parameters. Because multiple signatures (ie. overloading) is permitted, this method returns a list of signatures rather than a singleton. Signatures themselves are restricted to the top level parameters expected by a method. For instance if a method expects one array of structs as a parameter, and it returns a string, its signature is simply "string, array". If it expects three integers and returns a string, its signature is "string, int, int, int". If no signature is defined for the method, a non-array value is returned. In Python this means that the type of the returned value will be something other than list. .. method:: ServerProxy.system.methodHelp(name) This method takes one parameter, the name of a method implemented by the XML-RPC server. It returns a documentation string describing the use of that method. If no such string is available, an empty string is returned. The documentation string may contain HTML markup. .. versionchanged:: 3.5 Instances of :class:`ServerProxy` support the :term:`context manager` protocol for closing the underlying transport. A working example follows. The server code:: from xmlrpc.server import SimpleXMLRPCServer def is_even(n): return n % 2 == 0 server = SimpleXMLRPCServer(("localhost", 8000)) print("Listening on port 8000...") server.register_function(is_even, "is_even") server.serve_forever() The client code for the preceding server:: import xmlrpc.client with xmlrpc.client.ServerProxy("http://localhost:8000/") as proxy: print("3 is even: %s" % str(proxy.is_even(3))) print("100 is even: %s" % str(proxy.is_even(100))) .. _datetime-objects: DateTime Objects ---------------- .. class:: DateTime This class may be initialized with seconds since the epoch, a time tuple, an ISO 8601 time/date string, or a :class:`datetime.datetime` instance. It has the following methods, supported mainly for internal use by the marshalling/unmarshalling code: .. method:: decode(string) Accept a string as the instance's new time value. .. method:: encode(out) Write the XML-RPC encoding of this :class:`DateTime` item to the *out* stream object. It also supports certain of Python's built-in operators through rich comparison and :meth:`__repr__` methods. A working example follows. The server code:: import datetime from xmlrpc.server import SimpleXMLRPCServer import xmlrpc.client def today(): today = datetime.datetime.today() return xmlrpc.client.DateTime(today) server = SimpleXMLRPCServer(("localhost", 8000)) print("Listening on port 8000...") server.register_function(today, "today") server.serve_forever() The client code for the preceding server:: import xmlrpc.client import datetime proxy = xmlrpc.client.ServerProxy("http://localhost:8000/") today = proxy.today() # convert the ISO8601 string to a datetime object converted = datetime.datetime.strptime(today.value, "%Y%m%dT%H:%M:%S") print("Today: %s" % converted.strftime("%d.%m.%Y, %H:%M")) .. _binary-objects: Binary Objects -------------- .. class:: Binary This class may be initialized from bytes data (which may include NULs). The primary access to the content of a :class:`Binary` object is provided by an attribute: .. attribute:: data The binary data encapsulated by the :class:`Binary` instance. The data is provided as a :class:`bytes` object. :class:`Binary` objects have the following methods, supported mainly for internal use by the marshalling/unmarshalling code: .. method:: decode(bytes) Accept a base64 :class:`bytes` object and decode it as the instance's new data. .. method:: encode(out) Write the XML-RPC base 64 encoding of this binary item to the *out* stream object. The encoded data will have newlines every 76 characters as per :rfc:`RFC 2045 section 6.8 <2045#section-6.8>`, which was the de facto standard base64 specification when the XML-RPC spec was written. It also supports certain of Python's built-in operators through :meth:`__eq__` and :meth:`__ne__` methods. Example usage of the binary objects. We're going to transfer an image over XMLRPC:: from xmlrpc.server import SimpleXMLRPCServer import xmlrpc.client def python_logo(): with open("python_logo.jpg", "rb") as handle: return xmlrpc.client.Binary(handle.read()) server = SimpleXMLRPCServer(("localhost", 8000)) print("Listening on port 8000...") server.register_function(python_logo, 'python_logo') server.serve_forever() The client gets the image and saves it to a file:: import xmlrpc.client proxy = xmlrpc.client.ServerProxy("http://localhost:8000/") with open("fetched_python_logo.jpg", "wb") as handle: handle.write(proxy.python_logo().data) .. _fault-objects: Fault Objects ------------- .. class:: Fault A :class:`Fault` object encapsulates the content of an XML-RPC fault tag. Fault objects have the following attributes: .. attribute:: faultCode A string indicating the fault type. .. attribute:: faultString A string containing a diagnostic message associated with the fault. In the following example we're going to intentionally cause a :exc:`Fault` by returning a complex type object. The server code:: from xmlrpc.server import SimpleXMLRPCServer # A marshalling error is going to occur because we're returning a # complex number def add(x, y): return x+y+0j server = SimpleXMLRPCServer(("localhost", 8000)) print("Listening on port 8000...") server.register_function(add, 'add') server.serve_forever() The client code for the preceding server:: import xmlrpc.client proxy = xmlrpc.client.ServerProxy("http://localhost:8000/") try: proxy.add(2, 5) except xmlrpc.client.Fault as err: print("A fault occurred") print("Fault code: %d" % err.faultCode) print("Fault string: %s" % err.faultString) .. _protocol-error-objects: ProtocolError Objects --------------------- .. class:: ProtocolError A :class:`ProtocolError` object describes a protocol error in the underlying transport layer (such as a 404 'not found' error if the server named by the URI does not exist). It has the following attributes: .. attribute:: url The URI or URL that triggered the error. .. attribute:: errcode The error code. .. attribute:: errmsg The error message or diagnostic string. .. attribute:: headers A dict containing the headers of the HTTP/HTTPS request that triggered the error. In the following example we're going to intentionally cause a :exc:`ProtocolError` by providing an invalid URI:: import xmlrpc.client # create a ServerProxy with a URI that doesn't respond to XMLRPC requests proxy = xmlrpc.client.ServerProxy("http://google.com/") try: proxy.some_method() except xmlrpc.client.ProtocolError as err: print("A protocol error occurred") print("URL: %s" % err.url) print("HTTP/HTTPS headers: %s" % err.headers) print("Error code: %d" % err.errcode) print("Error message: %s" % err.errmsg) MultiCall Objects ----------------- The :class:`MultiCall` object provides a way to encapsulate multiple calls to a remote server into a single request [#]_. .. class:: MultiCall(server) Create an object used to boxcar method calls. *server* is the eventual target of the call. Calls can be made to the result object, but they will immediately return ``None``, and only store the call name and parameters in the :class:`MultiCall` object. Calling the object itself causes all stored calls to be transmitted as a single ``system.multicall`` request. The result of this call is a :term:`generator`; iterating over this generator yields the individual results. A usage example of this class follows. The server code:: from xmlrpc.server import SimpleXMLRPCServer def add(x, y): return x + y def subtract(x, y): return x - y def multiply(x, y): return x * y def divide(x, y): return x // y # A simple server with simple arithmetic functions server = SimpleXMLRPCServer(("localhost", 8000)) print("Listening on port 8000...") server.register_multicall_functions() server.register_function(add, 'add') server.register_function(subtract, 'subtract') server.register_function(multiply, 'multiply') server.register_function(divide, 'divide') server.serve_forever() The client code for the preceding server:: import xmlrpc.client proxy = xmlrpc.client.ServerProxy("http://localhost:8000/") multicall = xmlrpc.client.MultiCall(proxy) multicall.add(7, 3) multicall.subtract(7, 3) multicall.multiply(7, 3) multicall.divide(7, 3) result = multicall() print("7+3=%d, 7-3=%d, 7*3=%d, 7//3=%d" % tuple(result)) Convenience Functions --------------------- .. function:: dumps(params, methodname=None, methodresponse=None, encoding=None, allow_none=False) Convert *params* into an XML-RPC request. or into a response if *methodresponse* is true. *params* can be either a tuple of arguments or an instance of the :exc:`Fault` exception class. If *methodresponse* is true, only a single value can be returned, meaning that *params* must be of length 1. *encoding*, if supplied, is the encoding to use in the generated XML; the default is UTF-8. Python's :const:`None` value cannot be used in standard XML-RPC; to allow using it via an extension, provide a true value for *allow_none*. .. function:: loads(data, use_datetime=False, use_builtin_types=False) Convert an XML-RPC request or response into Python objects, a ``(params, methodname)``. *params* is a tuple of argument; *methodname* is a string, or ``None`` if no method name is present in the packet. If the XML-RPC packet represents a fault condition, this function will raise a :exc:`Fault` exception. The *use_builtin_types* flag can be used to cause date/time values to be presented as :class:`datetime.datetime` objects and binary data to be presented as :class:`bytes` objects; this flag is false by default. The obsolete *use_datetime* flag is similar to *use_builtin_types* but it applies only to date/time values. .. versionchanged:: 3.3 The *use_builtin_types* flag was added. .. _xmlrpc-client-example: Example of Client Usage ----------------------- :: # simple test program (from the XML-RPC specification) from xmlrpc.client import ServerProxy, Error # server = ServerProxy("http://localhost:8000") # local server with ServerProxy("http://betty.userland.com") as proxy: print(proxy) try: print(proxy.examples.getStateName(41)) except Error as v: print("ERROR", v) To access an XML-RPC server through a HTTP proxy, you need to define a custom transport. The following example shows how:: import http.client import xmlrpc.client class ProxiedTransport(xmlrpc.client.Transport): def set_proxy(self, host, port=None, headers=None): self.proxy = host, port self.proxy_headers = headers def make_connection(self, host): connection = http.client.HTTPConnection(*self.proxy) connection.set_tunnel(host, headers=self.proxy_headers) self._connection = host, connection return connection transport = ProxiedTransport() transport.set_proxy('proxy-server', 8080) server = xmlrpc.client.ServerProxy('http://betty.userland.com', transport=transport) print(server.examples.getStateName(41)) Example of Client and Server Usage ---------------------------------- See :ref:`simplexmlrpcserver-example`. .. rubric:: Footnotes .. [#] This approach has been first presented in `a discussion on xmlrpc.com <https://web.archive.org/web/20060624230303/http://www.xmlrpc.com/discuss/msgReader$1208?mode=topic>`_. .. the link now points to webarchive since the one at .. http://www.xmlrpc.com/discuss/msgReader%241208 is broken (and webadmin .. doesn't reply) PK����� 3]gP��P����faq/general.rst.txtnu�[��������:tocdepth: 2 ================== General Python FAQ ================== .. only:: html .. contents:: General Information =================== What is Python? --------------- Python is an interpreted, interactive, object-oriented programming language. It incorporates modules, exceptions, dynamic typing, very high level dynamic data types, and classes. Python combines remarkable power with very clear syntax. It has interfaces to many system calls and libraries, as well as to various window systems, and is extensible in C or C++. It is also usable as an extension language for applications that need a programmable interface. Finally, Python is portable: it runs on many Unix variants, on the Mac, and on Windows 2000 and later. To find out more, start with :ref:`tutorial-index`. The `Beginner's Guide to Python <https://wiki.python.org/moin/BeginnersGuide>`_ links to other introductory tutorials and resources for learning Python. What is the Python Software Foundation? --------------------------------------- The Python Software Foundation is an independent non-profit organization that holds the copyright on Python versions 2.1 and newer. The PSF's mission is to advance open source technology related to the Python programming language and to publicize the use of Python. The PSF's home page is at https://www.python.org/psf/. Donations to the PSF are tax-exempt in the US. If you use Python and find it helpful, please contribute via `the PSF donation page <https://www.python.org/psf/donations/>`_. Are there copyright restrictions on the use of Python? ------------------------------------------------------ You can do anything you want with the source, as long as you leave the copyrights in and display those copyrights in any documentation about Python that you produce. If you honor the copyright rules, it's OK to use Python for commercial use, to sell copies of Python in source or binary form (modified or unmodified), or to sell products that incorporate Python in some form. We would still like to know about all commercial use of Python, of course. See `the PSF license page <https://www.python.org/psf/license/>`_ to find further explanations and a link to the full text of the license. The Python logo is trademarked, and in certain cases permission is required to use it. Consult `the Trademark Usage Policy <https://www.python.org/psf/trademarks/>`__ for more information. Why was Python created in the first place? ------------------------------------------ Here's a *very* brief summary of what started it all, written by Guido van Rossum: I had extensive experience with implementing an interpreted language in the ABC group at CWI, and from working with this group I had learned a lot about language design. This is the origin of many Python features, including the use of indentation for statement grouping and the inclusion of very-high-level data types (although the details are all different in Python). I had a number of gripes about the ABC language, but also liked many of its features. It was impossible to extend the ABC language (or its implementation) to remedy my complaints -- in fact its lack of extensibility was one of its biggest problems. I had some experience with using Modula-2+ and talked with the designers of Modula-3 and read the Modula-3 report. Modula-3 is the origin of the syntax and semantics used for exceptions, and some other Python features. I was working in the Amoeba distributed operating system group at CWI. We needed a better way to do system administration than by writing either C programs or Bourne shell scripts, since Amoeba had its own system call interface which wasn't easily accessible from the Bourne shell. My experience with error handling in Amoeba made me acutely aware of the importance of exceptions as a programming language feature. It occurred to me that a scripting language with a syntax like ABC but with access to the Amoeba system calls would fill the need. I realized that it would be foolish to write an Amoeba-specific language, so I decided that I needed a language that was generally extensible. During the 1989 Christmas holidays, I had a lot of time on my hand, so I decided to give it a try. During the next year, while still mostly working on it in my own time, Python was used in the Amoeba project with increasing success, and the feedback from colleagues made me add many early improvements. In February 1991, after just over a year of development, I decided to post to USENET. The rest is in the ``Misc/HISTORY`` file. What is Python good for? ------------------------ Python is a high-level general-purpose programming language that can be applied to many different classes of problems. The language comes with a large standard library that covers areas such as string processing (regular expressions, Unicode, calculating differences between files), Internet protocols (HTTP, FTP, SMTP, XML-RPC, POP, IMAP, CGI programming), software engineering (unit testing, logging, profiling, parsing Python code), and operating system interfaces (system calls, filesystems, TCP/IP sockets). Look at the table of contents for :ref:`library-index` to get an idea of what's available. A wide variety of third-party extensions are also available. Consult `the Python Package Index <https://pypi.org>`_ to find packages of interest to you. How does the Python version numbering scheme work? -------------------------------------------------- Python versions are numbered A.B.C or A.B. A is the major version number -- it is only incremented for really major changes in the language. B is the minor version number, incremented for less earth-shattering changes. C is the micro-level -- it is incremented for each bugfix release. See :pep:`6` for more information about bugfix releases. Not all releases are bugfix releases. In the run-up to a new major release, a series of development releases are made, denoted as alpha, beta, or release candidate. Alphas are early releases in which interfaces aren't yet finalized; it's not unexpected to see an interface change between two alpha releases. Betas are more stable, preserving existing interfaces but possibly adding new modules, and release candidates are frozen, making no changes except as needed to fix critical bugs. Alpha, beta and release candidate versions have an additional suffix. The suffix for an alpha version is "aN" for some small number N, the suffix for a beta version is "bN" for some small number N, and the suffix for a release candidate version is "cN" for some small number N. In other words, all versions labeled 2.0aN precede the versions labeled 2.0bN, which precede versions labeled 2.0cN, and *those* precede 2.0. You may also find version numbers with a "+" suffix, e.g. "2.2+". These are unreleased versions, built directly from the CPython development repository. In practice, after a final minor release is made, the version is incremented to the next minor version, which becomes the "a0" version, e.g. "2.4a0". See also the documentation for :data:`sys.version`, :data:`sys.hexversion`, and :data:`sys.version_info`. How do I obtain a copy of the Python source? -------------------------------------------- The latest Python source distribution is always available from python.org, at https://www.python.org/downloads/. The latest development sources can be obtained at https://github.com/python/cpython/. The source distribution is a gzipped tar file containing the complete C source, Sphinx-formatted documentation, Python library modules, example programs, and several useful pieces of freely distributable software. The source will compile and run out of the box on most UNIX platforms. Consult the `Getting Started section of the Python Developer's Guide <https://devguide.python.org/setup/>`__ for more information on getting the source code and compiling it. How do I get documentation on Python? ------------------------------------- .. XXX mention py3k The standard documentation for the current stable version of Python is available at https://docs.python.org/3/. PDF, plain text, and downloadable HTML versions are also available at https://docs.python.org/3/download.html. The documentation is written in reStructuredText and processed by `the Sphinx documentation tool <http://sphinx-doc.org/>`__. The reStructuredText source for the documentation is part of the Python source distribution. I've never programmed before. Is there a Python tutorial? --------------------------------------------------------- There are numerous tutorials and books available. The standard documentation includes :ref:`tutorial-index`. Consult `the Beginner's Guide <https://wiki.python.org/moin/BeginnersGuide>`_ to find information for beginning Python programmers, including lists of tutorials. Is there a newsgroup or mailing list devoted to Python? ------------------------------------------------------- There is a newsgroup, :newsgroup:`comp.lang.python`, and a mailing list, `python-list <https://mail.python.org/mailman/listinfo/python-list>`_. The newsgroup and mailing list are gatewayed into each other -- if you can read news it's unnecessary to subscribe to the mailing list. :newsgroup:`comp.lang.python` is high-traffic, receiving hundreds of postings every day, and Usenet readers are often more able to cope with this volume. Announcements of new software releases and events can be found in comp.lang.python.announce, a low-traffic moderated list that receives about five postings per day. It's available as `the python-announce mailing list <https://mail.python.org/mailman/listinfo/python-announce-list>`_. More info about other mailing lists and newsgroups can be found at https://www.python.org/community/lists/. How do I get a beta test version of Python? ------------------------------------------- Alpha and beta releases are available from https://www.python.org/downloads/. All releases are announced on the comp.lang.python and comp.lang.python.announce newsgroups and on the Python home page at https://www.python.org/; an RSS feed of news is available. You can also access the development version of Python through Git. See `The Python Developer's Guide <https://devguide.python.org/>`_ for details. How do I submit bug reports and patches for Python? --------------------------------------------------- To report a bug or submit a patch, please use the Roundup installation at https://bugs.python.org/. You must have a Roundup account to report bugs; this makes it possible for us to contact you if we have follow-up questions. It will also enable Roundup to send you updates as we act on your bug. If you had previously used SourceForge to report bugs to Python, you can obtain your Roundup password through Roundup's `password reset procedure <https://bugs.python.org/user?@template=forgotten>`_. For more information on how Python is developed, consult `the Python Developer's Guide <https://devguide.python.org/>`_. Are there any published articles about Python that I can reference? ------------------------------------------------------------------- It's probably best to cite your favorite book about Python. The very first article about Python was written in 1991 and is now quite outdated. Guido van Rossum and Jelke de Boer, "Interactively Testing Remote Servers Using the Python Programming Language", CWI Quarterly, Volume 4, Issue 4 (December 1991), Amsterdam, pp 283--303. Are there any books on Python? ------------------------------ Yes, there are many, and more are being published. See the python.org wiki at https://wiki.python.org/moin/PythonBooks for a list. You can also search online bookstores for "Python" and filter out the Monty Python references; or perhaps search for "Python" and "language". Where in the world is www.python.org located? --------------------------------------------- The Python project's infrastructure is located all over the world. `www.python.org <https://www.python.org>`_ is graciously hosted by `Rackspace <https://www.rackspace.com>`_, with CDN caching provided by `Fastly <https://www.fastly.com>`_. `Upfront Systems <http://www.upfrontsystems.co.za/>`_ hosts `bugs.python.org <https://bugs.python.org>`_. Many other Python services like `the Wiki <https://wiki.python.org>`_ are hosted by `Oregon State University Open Source Lab <https://osuosl.org>`_. Why is it called Python? ------------------------ When he began implementing Python, Guido van Rossum was also reading the published scripts from `"Monty Python's Flying Circus" <https://en.wikipedia.org/wiki/Monty_Python>`__, a BBC comedy series from the 1970s. Van Rossum thought he needed a name that was short, unique, and slightly mysterious, so he decided to call the language Python. Do I have to like "Monty Python's Flying Circus"? ------------------------------------------------- No, but it helps. :) Python in the real world ======================== How stable is Python? --------------------- Very stable. New, stable releases have been coming out roughly every 6 to 18 months since 1991, and this seems likely to continue. Currently there are usually around 18 months between major releases. The developers issue "bugfix" releases of older versions, so the stability of existing releases gradually improves. Bugfix releases, indicated by a third component of the version number (e.g. 2.5.3, 2.6.2), are managed for stability; only fixes for known problems are included in a bugfix release, and it's guaranteed that interfaces will remain the same throughout a series of bugfix releases. The latest stable releases can always be found on the `Python download page <https://www.python.org/downloads/>`_. There are two recommended production-ready versions at this point in time, because at the moment there are two branches of stable releases: 2.x and 3.x. Python 3.x may be less useful than 2.x, since currently there is more third party software available for Python 2 than for Python 3. Python 2 code will generally not run unchanged in Python 3. How many people are using Python? --------------------------------- There are probably tens of thousands of users, though it's difficult to obtain an exact count. Python is available for free download, so there are no sales figures, and it's available from many different sites and packaged with many Linux distributions, so download statistics don't tell the whole story either. The comp.lang.python newsgroup is very active, but not all Python users post to the group or even read it. Have any significant projects been done in Python? -------------------------------------------------- See https://www.python.org/about/success for a list of projects that use Python. Consulting the proceedings for `past Python conferences <https://www.python.org/community/workshops/>`_ will reveal contributions from many different companies and organizations. High-profile Python projects include `the Mailman mailing list manager <http://www.list.org>`_ and `the Zope application server <http://www.zope.org>`_. Several Linux distributions, most notably `Red Hat <https://www.redhat.com>`_, have written part or all of their installer and system administration software in Python. Companies that use Python internally include Google, Yahoo, and Lucasfilm Ltd. What new developments are expected for Python in the future? ------------------------------------------------------------ See https://www.python.org/dev/peps/ for the Python Enhancement Proposals (PEPs). PEPs are design documents describing a suggested new feature for Python, providing a concise technical specification and a rationale. Look for a PEP titled "Python X.Y Release Schedule", where X.Y is a version that hasn't been publicly released yet. New development is discussed on `the python-dev mailing list <https://mail.python.org/mailman/listinfo/python-dev/>`_. Is it reasonable to propose incompatible changes to Python? ----------------------------------------------------------- In general, no. There are already millions of lines of Python code around the world, so any change in the language that invalidates more than a very small fraction of existing programs has to be frowned upon. Even if you can provide a conversion program, there's still the problem of updating all documentation; many books have been written about Python, and we don't want to invalidate them all at a single stroke. Providing a gradual upgrade path is necessary if a feature has to be changed. :pep:`5` describes the procedure followed for introducing backward-incompatible changes while minimizing disruption for users. Is Python a good language for beginning programmers? ---------------------------------------------------- Yes. It is still common to start students with a procedural and statically typed language such as Pascal, C, or a subset of C++ or Java. Students may be better served by learning Python as their first language. Python has a very simple and consistent syntax and a large standard library and, most importantly, using Python in a beginning programming course lets students concentrate on important programming skills such as problem decomposition and data type design. With Python, students can be quickly introduced to basic concepts such as loops and procedures. They can probably even work with user-defined objects in their very first course. For a student who has never programmed before, using a statically typed language seems unnatural. It presents additional complexity that the student must master and slows the pace of the course. The students are trying to learn to think like a computer, decompose problems, design consistent interfaces, and encapsulate data. While learning to use a statically typed language is important in the long term, it is not necessarily the best topic to address in the students' first programming course. Many other aspects of Python make it a good first language. Like Java, Python has a large standard library so that students can be assigned programming projects very early in the course that *do* something. Assignments aren't restricted to the standard four-function calculator and check balancing programs. By using the standard library, students can gain the satisfaction of working on realistic applications as they learn the fundamentals of programming. Using the standard library also teaches students about code reuse. Third-party modules such as PyGame are also helpful in extending the students' reach. Python's interactive interpreter enables students to test language features while they're programming. They can keep a window with the interpreter running while they enter their program's source in another window. If they can't remember the methods for a list, they can do something like this:: >>> L = [] >>> dir(L) # doctest: +NORMALIZE_WHITESPACE ['__add__', '__class__', '__contains__', '__delattr__', '__delitem__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', 'append', 'clear', 'copy', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort'] >>> [d for d in dir(L) if '__' not in d] ['append', 'clear', 'copy', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort'] >>> help(L.append) Help on built-in function append: <BLANKLINE> append(...) L.append(object) -> None -- append object to end <BLANKLINE> >>> L.append(1) >>> L [1] With the interpreter, documentation is never far from the student as they are programming. There are also good IDEs for Python. IDLE is a cross-platform IDE for Python that is written in Python using Tkinter. PythonWin is a Windows-specific IDE. Emacs users will be happy to know that there is a very good Python mode for Emacs. All of these programming environments provide syntax highlighting, auto-indenting, and access to the interactive interpreter while coding. Consult `the Python wiki <https://wiki.python.org/moin/PythonEditors>`_ for a full list of Python editing environments. If you want to discuss Python's use in education, you may be interested in joining `the edu-sig mailing list <https://www.python.org/community/sigs/current/edu-sig>`_. PK����� 3]ioa��a����faq/design.rst.txtnu�[��������====================== Design and History FAQ ====================== .. only:: html .. contents:: Why does Python use indentation for grouping of statements? ----------------------------------------------------------- Guido van Rossum believes that using indentation for grouping is extremely elegant and contributes a lot to the clarity of the average Python program. Most people learn to love this feature after a while. Since there are no begin/end brackets there cannot be a disagreement between grouping perceived by the parser and the human reader. Occasionally C programmers will encounter a fragment of code like this:: if (x <= y) x++; y--; z++; Only the ``x++`` statement is executed if the condition is true, but the indentation leads you to believe otherwise. Even experienced C programmers will sometimes stare at it a long time wondering why ``y`` is being decremented even for ``x > y``. Because there are no begin/end brackets, Python is much less prone to coding-style conflicts. In C there are many different ways to place the braces. If you're used to reading and writing code that uses one style, you will feel at least slightly uneasy when reading (or being required to write) another style. Many coding styles place begin/end brackets on a line by themselves. This makes programs considerably longer and wastes valuable screen space, making it harder to get a good overview of a program. Ideally, a function should fit on one screen (say, 20--30 lines). 20 lines of Python can do a lot more work than 20 lines of C. This is not solely due to the lack of begin/end brackets -- the lack of declarations and the high-level data types are also responsible -- but the indentation-based syntax certainly helps. Why am I getting strange results with simple arithmetic operations? ------------------------------------------------------------------- See the next question. Why are floating-point calculations so inaccurate? -------------------------------------------------- Users are often surprised by results like this:: >>> 1.2 - 1.0 0.19999999999999996 and think it is a bug in Python. It's not. This has little to do with Python, and much more to do with how the underlying platform handles floating-point numbers. The :class:`float` type in CPython uses a C ``double`` for storage. A :class:`float` object's value is stored in binary floating-point with a fixed precision (typically 53 bits) and Python uses C operations, which in turn rely on the hardware implementation in the processor, to perform floating-point operations. This means that as far as floating-point operations are concerned, Python behaves like many popular languages including C and Java. Many numbers that can be written easily in decimal notation cannot be expressed exactly in binary floating-point. For example, after:: >>> x = 1.2 the value stored for ``x`` is a (very good) approximation to the decimal value ``1.2``, but is not exactly equal to it. On a typical machine, the actual stored value is:: 1.0011001100110011001100110011001100110011001100110011 (binary) which is exactly:: 1.1999999999999999555910790149937383830547332763671875 (decimal) The typical precision of 53 bits provides Python floats with 15--16 decimal digits of accuracy. For a fuller explanation, please see the :ref:`floating point arithmetic <tut-fp-issues>` chapter in the Python tutorial. Why are Python strings immutable? --------------------------------- There are several advantages. One is performance: knowing that a string is immutable means we can allocate space for it at creation time, and the storage requirements are fixed and unchanging. This is also one of the reasons for the distinction between tuples and lists. Another advantage is that strings in Python are considered as "elemental" as numbers. No amount of activity will change the value 8 to anything else, and in Python, no amount of activity will change the string "eight" to anything else. .. _why-self: Why must 'self' be used explicitly in method definitions and calls? ------------------------------------------------------------------- The idea was borrowed from Modula-3. It turns out to be very useful, for a variety of reasons. First, it's more obvious that you are using a method or instance attribute instead of a local variable. Reading ``self.x`` or ``self.meth()`` makes it absolutely clear that an instance variable or method is used even if you don't know the class definition by heart. In C++, you can sort of tell by the lack of a local variable declaration (assuming globals are rare or easily recognizable) -- but in Python, there are no local variable declarations, so you'd have to look up the class definition to be sure. Some C++ and Java coding standards call for instance attributes to have an ``m_`` prefix, so this explicitness is still useful in those languages, too. Second, it means that no special syntax is necessary if you want to explicitly reference or call the method from a particular class. In C++, if you want to use a method from a base class which is overridden in a derived class, you have to use the ``::`` operator -- in Python you can write ``baseclass.methodname(self, <argument list>)``. This is particularly useful for :meth:`__init__` methods, and in general in cases where a derived class method wants to extend the base class method of the same name and thus has to call the base class method somehow. Finally, for instance variables it solves a syntactic problem with assignment: since local variables in Python are (by definition!) those variables to which a value is assigned in a function body (and that aren't explicitly declared global), there has to be some way to tell the interpreter that an assignment was meant to assign to an instance variable instead of to a local variable, and it should preferably be syntactic (for efficiency reasons). C++ does this through declarations, but Python doesn't have declarations and it would be a pity having to introduce them just for this purpose. Using the explicit ``self.var`` solves this nicely. Similarly, for using instance variables, having to write ``self.var`` means that references to unqualified names inside a method don't have to search the instance's directories. To put it another way, local variables and instance variables live in two different namespaces, and you need to tell Python which namespace to use. Why can't I use an assignment in an expression? ----------------------------------------------- Many people used to C or Perl complain that they want to use this C idiom: .. code-block:: c while (line = readline(f)) { // do something with line } where in Python you're forced to write this:: while True: line = f.readline() if not line: break ... # do something with line The reason for not allowing assignment in Python expressions is a common, hard-to-find bug in those other languages, caused by this construct: .. code-block:: c if (x = 0) { // error handling } else { // code that only works for nonzero x } The error is a simple typo: ``x = 0``, which assigns 0 to the variable ``x``, was written while the comparison ``x == 0`` is certainly what was intended. Many alternatives have been proposed. Most are hacks that save some typing but use arbitrary or cryptic syntax or keywords, and fail the simple criterion for language change proposals: it should intuitively suggest the proper meaning to a human reader who has not yet been introduced to the construct. An interesting phenomenon is that most experienced Python programmers recognize the ``while True`` idiom and don't seem to be missing the assignment in expression construct much; it's only newcomers who express a strong desire to add this to the language. There's an alternative way of spelling this that seems attractive but is generally less robust than the "while True" solution:: line = f.readline() while line: ... # do something with line... line = f.readline() The problem with this is that if you change your mind about exactly how you get the next line (e.g. you want to change it into ``sys.stdin.readline()``) you have to remember to change two places in your program -- the second occurrence is hidden at the bottom of the loop. The best approach is to use iterators, making it possible to loop through objects using the ``for`` statement. For example, :term:`file objects <file object>` support the iterator protocol, so you can write simply:: for line in f: ... # do something with line... Why does Python use methods for some functionality (e.g. list.index()) but functions for other (e.g. len(list))? ---------------------------------------------------------------------------------------------------------------- As Guido said: (a) For some operations, prefix notation just reads better than postfix -- prefix (and infix!) operations have a long tradition in mathematics which likes notations where the visuals help the mathematician thinking about a problem. Compare the easy with which we rewrite a formula like x*(a+b) into x*a + x*b to the clumsiness of doing the same thing using a raw OO notation. (b) When I read code that says len(x) I *know* that it is asking for the length of something. This tells me two things: the result is an integer, and the argument is some kind of container. To the contrary, when I read x.len(), I have to already know that x is some kind of container implementing an interface or inheriting from a class that has a standard len(). Witness the confusion we occasionally have when a class that is not implementing a mapping has a get() or keys() method, or something that isn't a file has a write() method. -- https://mail.python.org/pipermail/python-3000/2006-November/004643.html Why is join() a string method instead of a list or tuple method? ---------------------------------------------------------------- Strings became much more like other standard types starting in Python 1.6, when methods were added which give the same functionality that has always been available using the functions of the string module. Most of these new methods have been widely accepted, but the one which appears to make some programmers feel uncomfortable is:: ", ".join(['1', '2', '4', '8', '16']) which gives the result:: "1, 2, 4, 8, 16" There are two common arguments against this usage. The first runs along the lines of: "It looks really ugly using a method of a string literal (string constant)", to which the answer is that it might, but a string literal is just a fixed value. If the methods are to be allowed on names bound to strings there is no logical reason to make them unavailable on literals. The second objection is typically cast as: "I am really telling a sequence to join its members together with a string constant". Sadly, you aren't. For some reason there seems to be much less difficulty with having :meth:`~str.split` as a string method, since in that case it is easy to see that :: "1, 2, 4, 8, 16".split(", ") is an instruction to a string literal to return the substrings delimited by the given separator (or, by default, arbitrary runs of white space). :meth:`~str.join` is a string method because in using it you are telling the separator string to iterate over a sequence of strings and insert itself between adjacent elements. This method can be used with any argument which obeys the rules for sequence objects, including any new classes you might define yourself. Similar methods exist for bytes and bytearray objects. How fast are exceptions? ------------------------ A try/except block is extremely efficient if no exceptions are raised. Actually catching an exception is expensive. In versions of Python prior to 2.0 it was common to use this idiom:: try: value = mydict[key] except KeyError: mydict[key] = getvalue(key) value = mydict[key] This only made sense when you expected the dict to have the key almost all the time. If that wasn't the case, you coded it like this:: if key in mydict: value = mydict[key] else: value = mydict[key] = getvalue(key) For this specific case, you could also use ``value = dict.setdefault(key, getvalue(key))``, but only if the ``getvalue()`` call is cheap enough because it is evaluated in all cases. Why isn't there a switch or case statement in Python? ----------------------------------------------------- You can do this easily enough with a sequence of ``if... elif... elif... else``. There have been some proposals for switch statement syntax, but there is no consensus (yet) on whether and how to do range tests. See :pep:`275` for complete details and the current status. For cases where you need to choose from a very large number of possibilities, you can create a dictionary mapping case values to functions to call. For example:: def function_1(...): ... functions = {'a': function_1, 'b': function_2, 'c': self.method_1, ...} func = functions[value] func() For calling methods on objects, you can simplify yet further by using the :func:`getattr` built-in to retrieve methods with a particular name:: def visit_a(self, ...): ... ... def dispatch(self, value): method_name = 'visit_' + str(value) method = getattr(self, method_name) method() It's suggested that you use a prefix for the method names, such as ``visit_`` in this example. Without such a prefix, if values are coming from an untrusted source, an attacker would be able to call any method on your object. Can't you emulate threads in the interpreter instead of relying on an OS-specific thread implementation? -------------------------------------------------------------------------------------------------------- Answer 1: Unfortunately, the interpreter pushes at least one C stack frame for each Python stack frame. Also, extensions can call back into Python at almost random moments. Therefore, a complete threads implementation requires thread support for C. Answer 2: Fortunately, there is `Stackless Python <http://www.stackless.com>`_, which has a completely redesigned interpreter loop that avoids the C stack. Why can't lambda expressions contain statements? ------------------------------------------------ Python lambda expressions cannot contain statements because Python's syntactic framework can't handle statements nested inside expressions. However, in Python, this is not a serious problem. Unlike lambda forms in other languages, where they add functionality, Python lambdas are only a shorthand notation if you're too lazy to define a function. Functions are already first class objects in Python, and can be declared in a local scope. Therefore the only advantage of using a lambda instead of a locally-defined function is that you don't need to invent a name for the function -- but that's just a local variable to which the function object (which is exactly the same type of object that a lambda expression yields) is assigned! Can Python be compiled to machine code, C or some other language? ----------------------------------------------------------------- `Cython <http://cython.org/>`_ compiles a modified version of Python with optional annotations into C extensions. `Nuitka <http://www.nuitka.net/>`_ is an up-and-coming compiler of Python into C++ code, aiming to support the full Python language. For compiling to Java you can consider `VOC <https://voc.readthedocs.io>`_. How does Python manage memory? ------------------------------ The details of Python memory management depend on the implementation. The standard implementation of Python, :term:`CPython`, uses reference counting to detect inaccessible objects, and another mechanism to collect reference cycles, periodically executing a cycle detection algorithm which looks for inaccessible cycles and deletes the objects involved. The :mod:`gc` module provides functions to perform a garbage collection, obtain debugging statistics, and tune the collector's parameters. Other implementations (such as `Jython <http://www.jython.org>`_ or `PyPy <http://www.pypy.org>`_), however, can rely on a different mechanism such as a full-blown garbage collector. This difference can cause some subtle porting problems if your Python code depends on the behavior of the reference counting implementation. In some Python implementations, the following code (which is fine in CPython) will probably run out of file descriptors:: for file in very_long_list_of_files: f = open(file) c = f.read(1) Indeed, using CPython's reference counting and destructor scheme, each new assignment to *f* closes the previous file. With a traditional GC, however, those file objects will only get collected (and closed) at varying and possibly long intervals. If you want to write code that will work with any Python implementation, you should explicitly close the file or use the :keyword:`with` statement; this will work regardless of memory management scheme:: for file in very_long_list_of_files: with open(file) as f: c = f.read(1) Why doesn't CPython use a more traditional garbage collection scheme? --------------------------------------------------------------------- For one thing, this is not a C standard feature and hence it's not portable. (Yes, we know about the Boehm GC library. It has bits of assembler code for *most* common platforms, not for all of them, and although it is mostly transparent, it isn't completely transparent; patches are required to get Python to work with it.) Traditional GC also becomes a problem when Python is embedded into other applications. While in a standalone Python it's fine to replace the standard malloc() and free() with versions provided by the GC library, an application embedding Python may want to have its *own* substitute for malloc() and free(), and may not want Python's. Right now, CPython works with anything that implements malloc() and free() properly. Why isn't all memory freed when CPython exits? ---------------------------------------------- Objects referenced from the global namespaces of Python modules are not always deallocated when Python exits. This may happen if there are circular references. There are also certain bits of memory that are allocated by the C library that are impossible to free (e.g. a tool like Purify will complain about these). Python is, however, aggressive about cleaning up memory on exit and does try to destroy every single object. If you want to force Python to delete certain things on deallocation use the :mod:`atexit` module to run a function that will force those deletions. Why are there separate tuple and list data types? ------------------------------------------------- Lists and tuples, while similar in many respects, are generally used in fundamentally different ways. Tuples can be thought of as being similar to Pascal records or C structs; they're small collections of related data which may be of different types which are operated on as a group. For example, a Cartesian coordinate is appropriately represented as a tuple of two or three numbers. Lists, on the other hand, are more like arrays in other languages. They tend to hold a varying number of objects all of which have the same type and which are operated on one-by-one. For example, ``os.listdir('.')`` returns a list of strings representing the files in the current directory. Functions which operate on this output would generally not break if you added another file or two to the directory. Tuples are immutable, meaning that once a tuple has been created, you can't replace any of its elements with a new value. Lists are mutable, meaning that you can always change a list's elements. Only immutable elements can be used as dictionary keys, and hence only tuples and not lists can be used as keys. How are lists implemented in CPython? ------------------------------------- CPython's lists are really variable-length arrays, not Lisp-style linked lists. The implementation uses a contiguous array of references to other objects, and keeps a pointer to this array and the array's length in a list head structure. This makes indexing a list ``a[i]`` an operation whose cost is independent of the size of the list or the value of the index. When items are appended or inserted, the array of references is resized. Some cleverness is applied to improve the performance of appending items repeatedly; when the array must be grown, some extra space is allocated so the next few times don't require an actual resize. How are dictionaries implemented in CPython? -------------------------------------------- CPython's dictionaries are implemented as resizable hash tables. Compared to B-trees, this gives better performance for lookup (the most common operation by far) under most circumstances, and the implementation is simpler. Dictionaries work by computing a hash code for each key stored in the dictionary using the :func:`hash` built-in function. The hash code varies widely depending on the key and a per-process seed; for example, "Python" could hash to -539294296 while "python", a string that differs by a single bit, could hash to 1142331976. The hash code is then used to calculate a location in an internal array where the value will be stored. Assuming that you're storing keys that all have different hash values, this means that dictionaries take constant time -- O(1), in computer science notation -- to retrieve a key. It also means that no sorted order of the keys is maintained, and traversing the array as the ``.keys()`` and ``.items()`` do will output the dictionary's content in some arbitrary jumbled order that can change with every invocation of a program. Why must dictionary keys be immutable? -------------------------------------- The hash table implementation of dictionaries uses a hash value calculated from the key value to find the key. If the key were a mutable object, its value could change, and thus its hash could also change. But since whoever changes the key object can't tell that it was being used as a dictionary key, it can't move the entry around in the dictionary. Then, when you try to look up the same object in the dictionary it won't be found because its hash value is different. If you tried to look up the old value it wouldn't be found either, because the value of the object found in that hash bin would be different. If you want a dictionary indexed with a list, simply convert the list to a tuple first; the function ``tuple(L)`` creates a tuple with the same entries as the list ``L``. Tuples are immutable and can therefore be used as dictionary keys. Some unacceptable solutions that have been proposed: - Hash lists by their address (object ID). This doesn't work because if you construct a new list with the same value it won't be found; e.g.:: mydict = {[1, 2]: '12'} print(mydict[[1, 2]]) would raise a KeyError exception because the id of the ``[1, 2]`` used in the second line differs from that in the first line. In other words, dictionary keys should be compared using ``==``, not using :keyword:`is`. - Make a copy when using a list as a key. This doesn't work because the list, being a mutable object, could contain a reference to itself, and then the copying code would run into an infinite loop. - Allow lists as keys but tell the user not to modify them. This would allow a class of hard-to-track bugs in programs when you forgot or modified a list by accident. It also invalidates an important invariant of dictionaries: every value in ``d.keys()`` is usable as a key of the dictionary. - Mark lists as read-only once they are used as a dictionary key. The problem is that it's not just the top-level object that could change its value; you could use a tuple containing a list as a key. Entering anything as a key into a dictionary would require marking all objects reachable from there as read-only -- and again, self-referential objects could cause an infinite loop. There is a trick to get around this if you need to, but use it at your own risk: You can wrap a mutable structure inside a class instance which has both a :meth:`__eq__` and a :meth:`__hash__` method. You must then make sure that the hash value for all such wrapper objects that reside in a dictionary (or other hash based structure), remain fixed while the object is in the dictionary (or other structure). :: class ListWrapper: def __init__(self, the_list): self.the_list = the_list def __eq__(self, other): return self.the_list == other.the_list def __hash__(self): l = self.the_list result = 98767 - len(l)*555 for i, el in enumerate(l): try: result = result + (hash(el) % 9999999) * 1001 + i except Exception: result = (result % 7777777) + i * 333 return result Note that the hash computation is complicated by the possibility that some members of the list may be unhashable and also by the possibility of arithmetic overflow. Furthermore it must always be the case that if ``o1 == o2`` (ie ``o1.__eq__(o2) is True``) then ``hash(o1) == hash(o2)`` (ie, ``o1.__hash__() == o2.__hash__()``), regardless of whether the object is in a dictionary or not. If you fail to meet these restrictions dictionaries and other hash based structures will misbehave. In the case of ListWrapper, whenever the wrapper object is in a dictionary the wrapped list must not change to avoid anomalies. Don't do this unless you are prepared to think hard about the requirements and the consequences of not meeting them correctly. Consider yourself warned. Why doesn't list.sort() return the sorted list? ----------------------------------------------- In situations where performance matters, making a copy of the list just to sort it would be wasteful. Therefore, :meth:`list.sort` sorts the list in place. In order to remind you of that fact, it does not return the sorted list. This way, you won't be fooled into accidentally overwriting a list when you need a sorted copy but also need to keep the unsorted version around. If you want to return a new list, use the built-in :func:`sorted` function instead. This function creates a new list from a provided iterable, sorts it and returns it. For example, here's how to iterate over the keys of a dictionary in sorted order:: for key in sorted(mydict): ... # do whatever with mydict[key]... How do you specify and enforce an interface spec in Python? ----------------------------------------------------------- An interface specification for a module as provided by languages such as C++ and Java describes the prototypes for the methods and functions of the module. Many feel that compile-time enforcement of interface specifications helps in the construction of large programs. Python 2.6 adds an :mod:`abc` module that lets you define Abstract Base Classes (ABCs). You can then use :func:`isinstance` and :func:`issubclass` to check whether an instance or a class implements a particular ABC. The :mod:`collections.abc` module defines a set of useful ABCs such as :class:`~collections.abc.Iterable`, :class:`~collections.abc.Container`, and :class:`~collections.abc.MutableMapping`. For Python, many of the advantages of interface specifications can be obtained by an appropriate test discipline for components. There is also a tool, PyChecker, which can be used to find problems due to subclassing. A good test suite for a module can both provide a regression test and serve as a module interface specification and a set of examples. Many Python modules can be run as a script to provide a simple "self test." Even modules which use complex external interfaces can often be tested in isolation using trivial "stub" emulations of the external interface. The :mod:`doctest` and :mod:`unittest` modules or third-party test frameworks can be used to construct exhaustive test suites that exercise every line of code in a module. An appropriate testing discipline can help build large complex applications in Python as well as having interface specifications would. In fact, it can be better because an interface specification cannot test certain properties of a program. For example, the :meth:`append` method is expected to add new elements to the end of some internal list; an interface specification cannot test that your :meth:`append` implementation will actually do this correctly, but it's trivial to check this property in a test suite. Writing test suites is very helpful, and you might want to design your code with an eye to making it easily tested. One increasingly popular technique, test-directed development, calls for writing parts of the test suite first, before you write any of the actual code. Of course Python allows you to be sloppy and not write test cases at all. Why is there no goto? --------------------- You can use exceptions to provide a "structured goto" that even works across function calls. Many feel that exceptions can conveniently emulate all reasonable uses of the "go" or "goto" constructs of C, Fortran, and other languages. For example:: class label(Exception): pass # declare a label try: ... if condition: raise label() # goto label ... except label: # where to goto pass ... This doesn't allow you to jump into the middle of a loop, but that's usually considered an abuse of goto anyway. Use sparingly. Why can't raw strings (r-strings) end with a backslash? ------------------------------------------------------- More precisely, they can't end with an odd number of backslashes: the unpaired backslash at the end escapes the closing quote character, leaving an unterminated string. Raw strings were designed to ease creating input for processors (chiefly regular expression engines) that want to do their own backslash escape processing. Such processors consider an unmatched trailing backslash to be an error anyway, so raw strings disallow that. In return, they allow you to pass on the string quote character by escaping it with a backslash. These rules work well when r-strings are used for their intended purpose. If you're trying to build Windows pathnames, note that all Windows system calls accept forward slashes too:: f = open("/mydir/file.txt") # works fine! If you're trying to build a pathname for a DOS command, try e.g. one of :: dir = r"\this\is\my\dos\dir" "\\" dir = r"\this\is\my\dos\dir\ "[:-1] dir = "\\this\\is\\my\\dos\\dir\\" Why doesn't Python have a "with" statement for attribute assignments? --------------------------------------------------------------------- Python has a 'with' statement that wraps the execution of a block, calling code on the entrance and exit from the block. Some language have a construct that looks like this:: with obj: a = 1 # equivalent to obj.a = 1 total = total + 1 # obj.total = obj.total + 1 In Python, such a construct would be ambiguous. Other languages, such as Object Pascal, Delphi, and C++, use static types, so it's possible to know, in an unambiguous way, what member is being assigned to. This is the main point of static typing -- the compiler *always* knows the scope of every variable at compile time. Python uses dynamic types. It is impossible to know in advance which attribute will be referenced at runtime. Member attributes may be added or removed from objects on the fly. This makes it impossible to know, from a simple reading, what attribute is being referenced: a local one, a global one, or a member attribute? For instance, take the following incomplete snippet:: def foo(a): with a: print(x) The snippet assumes that "a" must have a member attribute called "x". However, there is nothing in Python that tells the interpreter this. What should happen if "a" is, let us say, an integer? If there is a global variable named "x", will it be used inside the with block? As you see, the dynamic nature of Python makes such choices much harder. The primary benefit of "with" and similar language features (reduction of code volume) can, however, easily be achieved in Python by assignment. Instead of:: function(args).mydict[index][index].a = 21 function(args).mydict[index][index].b = 42 function(args).mydict[index][index].c = 63 write this:: ref = function(args).mydict[index][index] ref.a = 21 ref.b = 42 ref.c = 63 This also has the side-effect of increasing execution speed because name bindings are resolved at run-time in Python, and the second version only needs to perform the resolution once. Why are colons required for the if/while/def/class statements? -------------------------------------------------------------- The colon is required primarily to enhance readability (one of the results of the experimental ABC language). Consider this:: if a == b print(a) versus :: if a == b: print(a) Notice how the second one is slightly easier to read. Notice further how a colon sets off the example in this FAQ answer; it's a standard usage in English. Another minor reason is that the colon makes it easier for editors with syntax highlighting; they can look for colons to decide when indentation needs to be increased instead of having to do a more elaborate parsing of the program text. Why does Python allow commas at the end of lists and tuples? ------------------------------------------------------------ Python lets you add a trailing comma at the end of lists, tuples, and dictionaries:: [1, 2, 3,] ('a', 'b', 'c',) d = { "A": [1, 5], "B": [6, 7], # last trailing comma is optional but good style } There are several reasons to allow this. When you have a literal value for a list, tuple, or dictionary spread across multiple lines, it's easier to add more elements because you don't have to remember to add a comma to the previous line. The lines can also be reordered without creating a syntax error. Accidentally omitting the comma can lead to errors that are hard to diagnose. For example:: x = [ "fee", "fie" "foo", "fum" ] This list looks like it has four elements, but it actually contains three: "fee", "fiefoo" and "fum". Always adding the comma avoids this source of error. Allowing the trailing comma may also make programmatic code generation easier. PK����� 3]H������faq/gui.rst.txtnu�[��������:tocdepth: 2 ========================== Graphic User Interface FAQ ========================== .. only:: html .. contents:: .. XXX need review for Python 3. General GUI Questions ===================== What platform-independent GUI toolkits exist for Python? ======================================================== Depending on what platform(s) you are aiming at, there are several. Some of them haven't been ported to Python 3 yet. At least `Tkinter`_ and `Qt`_ are known to be Python 3-compatible. .. XXX check links Tkinter ------- Standard builds of Python include an object-oriented interface to the Tcl/Tk widget set, called :ref:`tkinter <Tkinter>`. This is probably the easiest to install (since it comes included with most `binary distributions <https://www.python.org/downloads/>`_ of Python) and use. For more info about Tk, including pointers to the source, see the `Tcl/Tk home page <https://www.tcl.tk>`_. Tcl/Tk is fully portable to the Mac OS X, Windows, and Unix platforms. wxWidgets --------- wxWidgets (https://www.wxwidgets.org) is a free, portable GUI class library written in C++ that provides a native look and feel on a number of platforms, with Windows, Mac OS X, GTK, X11, all listed as current stable targets. Language bindings are available for a number of languages including Python, Perl, Ruby, etc. wxPython (http://www.wxpython.org) is the Python binding for wxwidgets. While it often lags slightly behind the official wxWidgets releases, it also offers a number of features via pure Python extensions that are not available in other language bindings. There is an active wxPython user and developer community. Both wxWidgets and wxPython are free, open source, software with permissive licences that allow their use in commercial products as well as in freeware or shareware. Qt --- There are bindings available for the Qt toolkit (using either `PyQt <https://riverbankcomputing.com/software/pyqt/intro>`_ or `PySide <https://wiki.qt.io/PySide>`_) and for KDE (`PyKDE4 <https://techbase.kde.org/Languages/Python/Using_PyKDE_4>`__). PyQt is currently more mature than PySide, but you must buy a PyQt license from `Riverbank Computing <https://www.riverbankcomputing.com/commercial/license-faq>`_ if you want to write proprietary applications. PySide is free for all applications. Qt 4.5 upwards is licensed under the LGPL license; also, commercial licenses are available from `The Qt Company <https://www.qt.io/licensing/>`_. Gtk+ ---- The `GObject introspection bindings <https://wiki.gnome.org/Projects/PyGObject>`_ for Python allow you to write GTK+ 3 applications. There is also a `Python GTK+ 3 Tutorial <https://python-gtk-3-tutorial.readthedocs.org/en/latest/>`_. The older PyGtk bindings for the `Gtk+ 2 toolkit <http://www.gtk.org>`_ have been implemented by James Henstridge; see <http://www.pygtk.org>. Kivy ---- `Kivy <https://kivy.org/>`_ is a cross-platform GUI library supporting both desktop operating systems (Windows, macOS, Linux) and mobile devices (Android, iOS). It is written in Python and Cython, and can use a range of windowing backends. Kivy is free and open source software distributed under the MIT license. FLTK ---- Python bindings for `the FLTK toolkit <http://www.fltk.org>`_, a simple yet powerful and mature cross-platform windowing system, are available from `the PyFLTK project <http://pyfltk.sourceforge.net>`_. OpenGL ------ For OpenGL bindings, see `PyOpenGL <http://pyopengl.sourceforge.net>`_. What platform-specific GUI toolkits exist for Python? ======================================================== By installing the `PyObjc Objective-C bridge <https://pythonhosted.org/pyobjc/>`_, Python programs can use Mac OS X's Cocoa libraries. :ref:`Pythonwin <windows-faq>` by Mark Hammond includes an interface to the Microsoft Foundation Classes and a Python programming environment that's written mostly in Python using the MFC classes. Tkinter questions ================= How do I freeze Tkinter applications? ------------------------------------- Freeze is a tool to create stand-alone applications. When freezing Tkinter applications, the applications will not be truly stand-alone, as the application will still need the Tcl and Tk libraries. One solution is to ship the application with the Tcl and Tk libraries, and point to them at run-time using the :envvar:`TCL_LIBRARY` and :envvar:`TK_LIBRARY` environment variables. To get truly stand-alone applications, the Tcl scripts that form the library have to be integrated into the application as well. One tool supporting that is SAM (stand-alone modules), which is part of the Tix distribution (http://tix.sourceforge.net/). Build Tix with SAM enabled, perform the appropriate call to :c:func:`Tclsam_init`, etc. inside Python's :file:`Modules/tkappinit.c`, and link with libtclsam and libtksam (you might include the Tix libraries as well). Can I have Tk events handled while waiting for I/O? --------------------------------------------------- On platforms other than Windows, yes, and you don't even need threads! But you'll have to restructure your I/O code a bit. Tk has the equivalent of Xt's :c:func:`XtAddInput()` call, which allows you to register a callback function which will be called from the Tk mainloop when I/O is possible on a file descriptor. See :ref:`tkinter-file-handlers`. I can't get key bindings to work in Tkinter: why? ------------------------------------------------- An often-heard complaint is that event handlers bound to events with the :meth:`bind` method don't get handled even when the appropriate key is pressed. The most common cause is that the widget to which the binding applies doesn't have "keyboard focus". Check out the Tk documentation for the focus command. Usually a widget is given the keyboard focus by clicking in it (but not for labels; see the takefocus option). PK����� 3]Lz��Lz����faq/library.rst.txtnu�[��������:tocdepth: 2 ========================= Library and Extension FAQ ========================= .. only:: html .. contents:: General Library Questions ========================= How do I find a module or application to perform task X? -------------------------------------------------------- Check :ref:`the Library Reference <library-index>` to see if there's a relevant standard library module. (Eventually you'll learn what's in the standard library and will be able to skip this step.) For third-party packages, search the `Python Package Index <https://pypi.org>`_ or try `Google <https://www.google.com>`_ or another Web search engine. Searching for "Python" plus a keyword or two for your topic of interest will usually find something helpful. Where is the math.py (socket.py, regex.py, etc.) source file? ------------------------------------------------------------- If you can't find a source file for a module it may be a built-in or dynamically loaded module implemented in C, C++ or other compiled language. In this case you may not have the source file or it may be something like :file:`mathmodule.c`, somewhere in a C source directory (not on the Python Path). There are (at least) three kinds of modules in Python: 1) modules written in Python (.py); 2) modules written in C and dynamically loaded (.dll, .pyd, .so, .sl, etc); 3) modules written in C and linked with the interpreter; to get a list of these, type:: import sys print(sys.builtin_module_names) How do I make a Python script executable on Unix? ------------------------------------------------- You need to do two things: the script file's mode must be executable and the first line must begin with ``#!`` followed by the path of the Python interpreter. The first is done by executing ``chmod +x scriptfile`` or perhaps ``chmod 755 scriptfile``. The second can be done in a number of ways. The most straightforward way is to write :: #!/usr/local/bin/python as the very first line of your file, using the pathname for where the Python interpreter is installed on your platform. If you would like the script to be independent of where the Python interpreter lives, you can use the :program:`env` program. Almost all Unix variants support the following, assuming the Python interpreter is in a directory on the user's :envvar:`PATH`:: #!/usr/bin/env python *Don't* do this for CGI scripts. The :envvar:`PATH` variable for CGI scripts is often very minimal, so you need to use the actual absolute pathname of the interpreter. Occasionally, a user's environment is so full that the :program:`/usr/bin/env` program fails; or there's no env program at all. In that case, you can try the following hack (due to Alex Rezinsky): .. code-block:: sh #! /bin/sh """:" exec python $0 ${1+"$@"} """ The minor disadvantage is that this defines the script's __doc__ string. However, you can fix that by adding :: __doc__ = """...Whatever...""" Is there a curses/termcap package for Python? --------------------------------------------- .. XXX curses *is* built by default, isn't it? For Unix variants: The standard Python source distribution comes with a curses module in the :source:`Modules` subdirectory, though it's not compiled by default. (Note that this is not available in the Windows distribution -- there is no curses module for Windows.) The :mod:`curses` module supports basic curses features as well as many additional functions from ncurses and SYSV curses such as colour, alternative character set support, pads, and mouse support. This means the module isn't compatible with operating systems that only have BSD curses, but there don't seem to be any currently maintained OSes that fall into this category. For Windows: use `the consolelib module <http://effbot.org/zone/console-index.htm>`_. Is there an equivalent to C's onexit() in Python? ------------------------------------------------- The :mod:`atexit` module provides a register function that is similar to C's :c:func:`onexit`. Why don't my signal handlers work? ---------------------------------- The most common problem is that the signal handler is declared with the wrong argument list. It is called as :: handler(signum, frame) so it should be declared with two arguments:: def handler(signum, frame): ... Common tasks ============ How do I test a Python program or component? -------------------------------------------- Python comes with two testing frameworks. The :mod:`doctest` module finds examples in the docstrings for a module and runs them, comparing the output with the expected output given in the docstring. The :mod:`unittest` module is a fancier testing framework modelled on Java and Smalltalk testing frameworks. To make testing easier, you should use good modular design in your program. Your program should have almost all functionality encapsulated in either functions or class methods -- and this sometimes has the surprising and delightful effect of making the program run faster (because local variable accesses are faster than global accesses). Furthermore the program should avoid depending on mutating global variables, since this makes testing much more difficult to do. The "global main logic" of your program may be as simple as :: if __name__ == "__main__": main_logic() at the bottom of the main module of your program. Once your program is organized as a tractable collection of functions and class behaviours you should write test functions that exercise the behaviours. A test suite that automates a sequence of tests can be associated with each module. This sounds like a lot of work, but since Python is so terse and flexible it's surprisingly easy. You can make coding much more pleasant and fun by writing your test functions in parallel with the "production code", since this makes it easy to find bugs and even design flaws earlier. "Support modules" that are not intended to be the main module of a program may include a self-test of the module. :: if __name__ == "__main__": self_test() Even programs that interact with complex external interfaces may be tested when the external interfaces are unavailable by using "fake" interfaces implemented in Python. How do I create documentation from doc strings? ----------------------------------------------- The :mod:`pydoc` module can create HTML from the doc strings in your Python source code. An alternative for creating API documentation purely from docstrings is `epydoc <http://epydoc.sourceforge.net/>`_. `Sphinx <http://sphinx-doc.org>`_ can also include docstring content. How do I get a single keypress at a time? ----------------------------------------- For Unix variants there are several solutions. It's straightforward to do this using curses, but curses is a fairly large module to learn. .. XXX this doesn't work out of the box, some IO expert needs to check why Here's a solution without curses:: import termios, fcntl, sys, os fd = sys.stdin.fileno() oldterm = termios.tcgetattr(fd) newattr = termios.tcgetattr(fd) newattr[3] = newattr[3] & ~termios.ICANON & ~termios.ECHO termios.tcsetattr(fd, termios.TCSANOW, newattr) oldflags = fcntl.fcntl(fd, fcntl.F_GETFL) fcntl.fcntl(fd, fcntl.F_SETFL, oldflags | os.O_NONBLOCK) try: while True: try: c = sys.stdin.read(1) print("Got character", repr(c)) except OSError: pass finally: termios.tcsetattr(fd, termios.TCSAFLUSH, oldterm) fcntl.fcntl(fd, fcntl.F_SETFL, oldflags) You need the :mod:`termios` and the :mod:`fcntl` module for any of this to work, and I've only tried it on Linux, though it should work elsewhere. In this code, characters are read and printed one at a time. :func:`termios.tcsetattr` turns off stdin's echoing and disables canonical mode. :func:`fcntl.fnctl` is used to obtain stdin's file descriptor flags and modify them for non-blocking mode. Since reading stdin when it is empty results in an :exc:`OSError`, this error is caught and ignored. .. versionchanged:: 3.3 *sys.stdin.read* used to raise :exc:`IOError`. Starting from Python 3.3 :exc:`IOError` is alias for :exc:`OSError`. Threads ======= How do I program using threads? ------------------------------- Be sure to use the :mod:`threading` module and not the :mod:`_thread` module. The :mod:`threading` module builds convenient abstractions on top of the low-level primitives provided by the :mod:`_thread` module. Aahz has a set of slides from his threading tutorial that are helpful; see http://www.pythoncraft.com/OSCON2001/. None of my threads seem to run: why? ------------------------------------ As soon as the main thread exits, all threads are killed. Your main thread is running too quickly, giving the threads no time to do any work. A simple fix is to add a sleep to the end of the program that's long enough for all the threads to finish:: import threading, time def thread_task(name, n): for i in range(n): print(name, i) for i in range(10): T = threading.Thread(target=thread_task, args=(str(i), i)) T.start() time.sleep(10) # <---------------------------! But now (on many platforms) the threads don't run in parallel, but appear to run sequentially, one at a time! The reason is that the OS thread scheduler doesn't start a new thread until the previous thread is blocked. A simple fix is to add a tiny sleep to the start of the run function:: def thread_task(name, n): time.sleep(0.001) # <--------------------! for i in range(n): print(name, i) for i in range(10): T = threading.Thread(target=thread_task, args=(str(i), i)) T.start() time.sleep(10) Instead of trying to guess a good delay value for :func:`time.sleep`, it's better to use some kind of semaphore mechanism. One idea is to use the :mod:`queue` module to create a queue object, let each thread append a token to the queue when it finishes, and let the main thread read as many tokens from the queue as there are threads. How do I parcel out work among a bunch of worker threads? --------------------------------------------------------- The easiest way is to use the new :mod:`concurrent.futures` module, especially the :mod:`~concurrent.futures.ThreadPoolExecutor` class. Or, if you want fine control over the dispatching algorithm, you can write your own logic manually. Use the :mod:`queue` module to create a queue containing a list of jobs. The :class:`~queue.Queue` class maintains a list of objects and has a ``.put(obj)`` method that adds items to the queue and a ``.get()`` method to return them. The class will take care of the locking necessary to ensure that each job is handed out exactly once. Here's a trivial example:: import threading, queue, time # The worker thread gets jobs off the queue. When the queue is empty, it # assumes there will be no more work and exits. # (Realistically workers will run until terminated.) def worker(): print('Running worker') time.sleep(0.1) while True: try: arg = q.get(block=False) except queue.Empty: print('Worker', threading.currentThread(), end=' ') print('queue empty') break else: print('Worker', threading.currentThread(), end=' ') print('running with argument', arg) time.sleep(0.5) # Create queue q = queue.Queue() # Start a pool of 5 workers for i in range(5): t = threading.Thread(target=worker, name='worker %i' % (i+1)) t.start() # Begin adding work to the queue for i in range(50): q.put(i) # Give threads time to run print('Main thread sleeping') time.sleep(5) When run, this will produce the following output: .. code-block:: none Running worker Running worker Running worker Running worker Running worker Main thread sleeping Worker <Thread(worker 1, started 130283832797456)> running with argument 0 Worker <Thread(worker 2, started 130283824404752)> running with argument 1 Worker <Thread(worker 3, started 130283816012048)> running with argument 2 Worker <Thread(worker 4, started 130283807619344)> running with argument 3 Worker <Thread(worker 5, started 130283799226640)> running with argument 4 Worker <Thread(worker 1, started 130283832797456)> running with argument 5 ... Consult the module's documentation for more details; the :class:`~queue.Queue` class provides a featureful interface. What kinds of global value mutation are thread-safe? ---------------------------------------------------- A :term:`global interpreter lock` (GIL) is used internally to ensure that only one thread runs in the Python VM at a time. In general, Python offers to switch among threads only between bytecode instructions; how frequently it switches can be set via :func:`sys.setswitchinterval`. Each bytecode instruction and therefore all the C implementation code reached from each instruction is therefore atomic from the point of view of a Python program. In theory, this means an exact accounting requires an exact understanding of the PVM bytecode implementation. In practice, it means that operations on shared variables of built-in data types (ints, lists, dicts, etc) that "look atomic" really are. For example, the following operations are all atomic (L, L1, L2 are lists, D, D1, D2 are dicts, x, y are objects, i, j are ints):: L.append(x) L1.extend(L2) x = L[i] x = L.pop() L1[i:j] = L2 L.sort() x = y x.field = y D[x] = y D1.update(D2) D.keys() These aren't:: i = i+1 L.append(L[-1]) L[i] = L[j] D[x] = D[x] + 1 Operations that replace other objects may invoke those other objects' :meth:`__del__` method when their reference count reaches zero, and that can affect things. This is especially true for the mass updates to dictionaries and lists. When in doubt, use a mutex! Can't we get rid of the Global Interpreter Lock? ------------------------------------------------ .. XXX link to dbeazley's talk about GIL? The :term:`global interpreter lock` (GIL) is often seen as a hindrance to Python's deployment on high-end multiprocessor server machines, because a multi-threaded Python program effectively only uses one CPU, due to the insistence that (almost) all Python code can only run while the GIL is held. Back in the days of Python 1.5, Greg Stein actually implemented a comprehensive patch set (the "free threading" patches) that removed the GIL and replaced it with fine-grained locking. Adam Olsen recently did a similar experiment in his `python-safethread <http://code.google.com/p/python-safethread/>`_ project. Unfortunately, both experiments exhibited a sharp drop in single-thread performance (at least 30% slower), due to the amount of fine-grained locking necessary to compensate for the removal of the GIL. This doesn't mean that you can't make good use of Python on multi-CPU machines! You just have to be creative with dividing the work up between multiple *processes* rather than multiple *threads*. The :class:`~concurrent.futures.ProcessPoolExecutor` class in the new :mod:`concurrent.futures` module provides an easy way of doing so; the :mod:`multiprocessing` module provides a lower-level API in case you want more control over dispatching of tasks. Judicious use of C extensions will also help; if you use a C extension to perform a time-consuming task, the extension can release the GIL while the thread of execution is in the C code and allow other threads to get some work done. Some standard library modules such as :mod:`zlib` and :mod:`hashlib` already do this. It has been suggested that the GIL should be a per-interpreter-state lock rather than truly global; interpreters then wouldn't be able to share objects. Unfortunately, this isn't likely to happen either. It would be a tremendous amount of work, because many object implementations currently have global state. For example, small integers and short strings are cached; these caches would have to be moved to the interpreter state. Other object types have their own free list; these free lists would have to be moved to the interpreter state. And so on. And I doubt that it can even be done in finite time, because the same problem exists for 3rd party extensions. It is likely that 3rd party extensions are being written at a faster rate than you can convert them to store all their global state in the interpreter state. And finally, once you have multiple interpreters not sharing any state, what have you gained over running each interpreter in a separate process? Input and Output ================ How do I delete a file? (And other file questions...) ----------------------------------------------------- Use ``os.remove(filename)`` or ``os.unlink(filename)``; for documentation, see the :mod:`os` module. The two functions are identical; :func:`~os.unlink` is simply the name of the Unix system call for this function. To remove a directory, use :func:`os.rmdir`; use :func:`os.mkdir` to create one. ``os.makedirs(path)`` will create any intermediate directories in ``path`` that don't exist. ``os.removedirs(path)`` will remove intermediate directories as long as they're empty; if you want to delete an entire directory tree and its contents, use :func:`shutil.rmtree`. To rename a file, use ``os.rename(old_path, new_path)``. To truncate a file, open it using ``f = open(filename, "rb+")``, and use ``f.truncate(offset)``; offset defaults to the current seek position. There's also ``os.ftruncate(fd, offset)`` for files opened with :func:`os.open`, where *fd* is the file descriptor (a small integer). The :mod:`shutil` module also contains a number of functions to work on files including :func:`~shutil.copyfile`, :func:`~shutil.copytree`, and :func:`~shutil.rmtree`. How do I copy a file? --------------------- The :mod:`shutil` module contains a :func:`~shutil.copyfile` function. Note that on MacOS 9 it doesn't copy the resource fork and Finder info. How do I read (or write) binary data? ------------------------------------- To read or write complex binary data formats, it's best to use the :mod:`struct` module. It allows you to take a string containing binary data (usually numbers) and convert it to Python objects; and vice versa. For example, the following code reads two 2-byte integers and one 4-byte integer in big-endian format from a file:: import struct with open(filename, "rb") as f: s = f.read(8) x, y, z = struct.unpack(">hhl", s) The '>' in the format string forces big-endian data; the letter 'h' reads one "short integer" (2 bytes), and 'l' reads one "long integer" (4 bytes) from the string. For data that is more regular (e.g. a homogeneous list of ints or floats), you can also use the :mod:`array` module. .. note:: To read and write binary data, it is mandatory to open the file in binary mode (here, passing ``"rb"`` to :func:`open`). If you use ``"r"`` instead (the default), the file will be open in text mode and ``f.read()`` will return :class:`str` objects rather than :class:`bytes` objects. I can't seem to use os.read() on a pipe created with os.popen(); why? --------------------------------------------------------------------- :func:`os.read` is a low-level function which takes a file descriptor, a small integer representing the opened file. :func:`os.popen` creates a high-level file object, the same type returned by the built-in :func:`open` function. Thus, to read *n* bytes from a pipe *p* created with :func:`os.popen`, you need to use ``p.read(n)``. .. XXX update to use subprocess. See the :ref:`subprocess-replacements` section. How do I run a subprocess with pipes connected to both input and output? ------------------------------------------------------------------------ Use the :mod:`popen2` module. For example:: import popen2 fromchild, tochild = popen2.popen2("command") tochild.write("input\n") tochild.flush() output = fromchild.readline() Warning: in general it is unwise to do this because you can easily cause a deadlock where your process is blocked waiting for output from the child while the child is blocked waiting for input from you. This can be caused by the parent expecting the child to output more text than it does or by data being stuck in stdio buffers due to lack of flushing. The Python parent can of course explicitly flush the data it sends to the child before it reads any output, but if the child is a naive C program it may have been written to never explicitly flush its output, even if it is interactive, since flushing is normally automatic. Note that a deadlock is also possible if you use :func:`popen3` to read stdout and stderr. If one of the two is too large for the internal buffer (increasing the buffer size does not help) and you ``read()`` the other one first, there is a deadlock, too. Note on a bug in popen2: unless your program calls ``wait()`` or ``waitpid()``, finished child processes are never removed, and eventually calls to popen2 will fail because of a limit on the number of child processes. Calling :func:`os.waitpid` with the :data:`os.WNOHANG` option can prevent this; a good place to insert such a call would be before calling ``popen2`` again. In many cases, all you really need is to run some data through a command and get the result back. Unless the amount of data is very large, the easiest way to do this is to write it to a temporary file and run the command with that temporary file as input. The standard module :mod:`tempfile` exports a :func:`~tempfile.mktemp` function to generate unique temporary file names. :: import tempfile import os class Popen3: """ This is a deadlock-safe version of popen that returns an object with errorlevel, out (a string) and err (a string). (capturestderr may not work under windows.) Example: print(Popen3('grep spam','\n\nhere spam\n\n').out) """ def __init__(self,command,input=None,capturestderr=None): outfile=tempfile.mktemp() command="( %s ) > %s" % (command,outfile) if input: infile=tempfile.mktemp() open(infile,"w").write(input) command=command+" <"+infile if capturestderr: errfile=tempfile.mktemp() command=command+" 2>"+errfile self.errorlevel=os.system(command) >> 8 self.out=open(outfile,"r").read() os.remove(outfile) if input: os.remove(infile) if capturestderr: self.err=open(errfile,"r").read() os.remove(errfile) Note that many interactive programs (e.g. vi) don't work well with pipes substituted for standard input and output. You will have to use pseudo ttys ("ptys") instead of pipes. Or you can use a Python interface to Don Libes' "expect" library. A Python extension that interfaces to expect is called "expy" and available from http://expectpy.sourceforge.net. A pure Python solution that works like expect is `pexpect <https://pypi.org/project/pexpect/>`_. How do I access the serial (RS232) port? ---------------------------------------- For Win32, POSIX (Linux, BSD, etc.), Jython: http://pyserial.sourceforge.net For Unix, see a Usenet post by Mitch Chapman: https://groups.google.com/groups?selm=34A04430.CF9@ohioee.com Why doesn't closing sys.stdout (stdin, stderr) really close it? --------------------------------------------------------------- Python :term:`file objects <file object>` are a high-level layer of abstraction on low-level C file descriptors. For most file objects you create in Python via the built-in :func:`open` function, ``f.close()`` marks the Python file object as being closed from Python's point of view, and also arranges to close the underlying C file descriptor. This also happens automatically in ``f``'s destructor, when ``f`` becomes garbage. But stdin, stdout and stderr are treated specially by Python, because of the special status also given to them by C. Running ``sys.stdout.close()`` marks the Python-level file object as being closed, but does *not* close the associated C file descriptor. To close the underlying C file descriptor for one of these three, you should first be sure that's what you really want to do (e.g., you may confuse extension modules trying to do I/O). If it is, use :func:`os.close`:: os.close(stdin.fileno()) os.close(stdout.fileno()) os.close(stderr.fileno()) Or you can use the numeric constants 0, 1 and 2, respectively. Network/Internet Programming ============================ What WWW tools are there for Python? ------------------------------------ See the chapters titled :ref:`internet` and :ref:`netdata` in the Library Reference Manual. Python has many modules that will help you build server-side and client-side web systems. .. XXX check if wiki page is still up to date A summary of available frameworks is maintained by Paul Boddie at https://wiki.python.org/moin/WebProgramming\ . Cameron Laird maintains a useful set of pages about Python web technologies at http://phaseit.net/claird/comp.lang.python/web_python. How can I mimic CGI form submission (METHOD=POST)? -------------------------------------------------- I would like to retrieve web pages that are the result of POSTing a form. Is there existing code that would let me do this easily? Yes. Here's a simple example that uses urllib.request:: #!/usr/local/bin/python import urllib.request # build the query string qs = "First=Josephine&MI=Q&Last=Public" # connect and send the server a path req = urllib.request.urlopen('http://www.some-server.out-there' '/cgi-bin/some-cgi-script', data=qs) with req: msg, hdrs = req.read(), req.info() Note that in general for percent-encoded POST operations, query strings must be quoted using :func:`urllib.parse.urlencode`. For example, to send ``name=Guy Steele, Jr.``:: >>> import urllib.parse >>> urllib.parse.urlencode({'name': 'Guy Steele, Jr.'}) 'name=Guy+Steele%2C+Jr.' .. seealso:: :ref:`urllib-howto` for extensive examples. What module should I use to help with generating HTML? ------------------------------------------------------ .. XXX add modern template languages You can find a collection of useful links on the `Web Programming wiki page <https://wiki.python.org/moin/WebProgramming>`_. How do I send mail from a Python script? ---------------------------------------- Use the standard library module :mod:`smtplib`. Here's a very simple interactive mail sender that uses it. This method will work on any host that supports an SMTP listener. :: import sys, smtplib fromaddr = input("From: ") toaddrs = input("To: ").split(',') print("Enter message, end with ^D:") msg = '' while True: line = sys.stdin.readline() if not line: break msg += line # The actual mail send server = smtplib.SMTP('localhost') server.sendmail(fromaddr, toaddrs, msg) server.quit() A Unix-only alternative uses sendmail. The location of the sendmail program varies between systems; sometimes it is ``/usr/lib/sendmail``, sometimes ``/usr/sbin/sendmail``. The sendmail manual page will help you out. Here's some sample code:: import os SENDMAIL = "/usr/sbin/sendmail" # sendmail location p = os.popen("%s -t -i" % SENDMAIL, "w") p.write("To: receiver@example.com\n") p.write("Subject: test\n") p.write("\n") # blank line separating headers from body p.write("Some text\n") p.write("some more text\n") sts = p.close() if sts != 0: print("Sendmail exit status", sts) How do I avoid blocking in the connect() method of a socket? ------------------------------------------------------------ The :mod:`select` module is commonly used to help with asynchronous I/O on sockets. To prevent the TCP connect from blocking, you can set the socket to non-blocking mode. Then when you do the ``connect()``, you will either connect immediately (unlikely) or get an exception that contains the error number as ``.errno``. ``errno.EINPROGRESS`` indicates that the connection is in progress, but hasn't finished yet. Different OSes will return different values, so you're going to have to check what's returned on your system. You can use the ``connect_ex()`` method to avoid creating an exception. It will just return the errno value. To poll, you can call ``connect_ex()`` again later -- ``0`` or ``errno.EISCONN`` indicate that you're connected -- or you can pass this socket to select to check if it's writable. .. note:: The :mod:`asyncore` module presents a framework-like approach to the problem of writing non-blocking networking code. The third-party `Twisted <https://twistedmatrix.com/trac/>`_ library is a popular and feature-rich alternative. Databases ========= Are there any interfaces to database packages in Python? -------------------------------------------------------- Yes. Interfaces to disk-based hashes such as :mod:`DBM <dbm.ndbm>` and :mod:`GDBM <dbm.gnu>` are also included with standard Python. There is also the :mod:`sqlite3` module, which provides a lightweight disk-based relational database. Support for most relational databases is available. See the `DatabaseProgramming wiki page <https://wiki.python.org/moin/DatabaseProgramming>`_ for details. How do you implement persistent objects in Python? -------------------------------------------------- The :mod:`pickle` library module solves this in a very general way (though you still can't store things like open files, sockets or windows), and the :mod:`shelve` library module uses pickle and (g)dbm to create persistent mappings containing arbitrary Python objects. Mathematics and Numerics ======================== How do I generate random numbers in Python? ------------------------------------------- The standard module :mod:`random` implements a random number generator. Usage is simple:: import random random.random() This returns a random floating point number in the range [0, 1). There are also many other specialized generators in this module, such as: * ``randrange(a, b)`` chooses an integer in the range [a, b). * ``uniform(a, b)`` chooses a floating point number in the range [a, b). * ``normalvariate(mean, sdev)`` samples the normal (Gaussian) distribution. Some higher-level functions operate on sequences directly, such as: * ``choice(S)`` chooses random element from a given sequence * ``shuffle(L)`` shuffles a list in-place, i.e. permutes it randomly There's also a ``Random`` class you can instantiate to create independent multiple random number generators. PK����� 3]Y;- �� ����faq/installed.rst.txtnu�[��������============================================= "Why is Python Installed on my Computer?" FAQ ============================================= What is Python? --------------- Python is a programming language. It's used for many different applications. It's used in some high schools and colleges as an introductory programming language because Python is easy to learn, but it's also used by professional software developers at places such as Google, NASA, and Lucasfilm Ltd. If you wish to learn more about Python, start with the `Beginner's Guide to Python <https://wiki.python.org/moin/BeginnersGuide>`_. Why is Python installed on my machine? -------------------------------------- If you find Python installed on your system but don't remember installing it, there are several possible ways it could have gotten there. * Perhaps another user on the computer wanted to learn programming and installed it; you'll have to figure out who's been using the machine and might have installed it. * A third-party application installed on the machine might have been written in Python and included a Python installation. There are many such applications, from GUI programs to network servers and administrative scripts. * Some Windows machines also have Python installed. At this writing we're aware of computers from Hewlett-Packard and Compaq that include Python. Apparently some of HP/Compaq's administrative tools are written in Python. * Many Unix-compatible operating systems, such as Mac OS X and some Linux distributions, have Python installed by default; it's included in the base installation. Can I delete Python? -------------------- That depends on where Python came from. If someone installed it deliberately, you can remove it without hurting anything. On Windows, use the Add/Remove Programs icon in the Control Panel. If Python was installed by a third-party application, you can also remove it, but that application will no longer work. You should use that application's uninstaller rather than removing Python directly. If Python came with your operating system, removing it is not recommended. If you remove it, whatever tools were written in Python will no longer run, and some of them might be important to you. Reinstalling the whole system would then be required to fix things again. PK����� 3]x]<��]<����faq/windows.rst.txtnu�[��������:tocdepth: 2 .. highlightlang:: none .. _windows-faq: ===================== Python on Windows FAQ ===================== .. only:: html .. contents:: .. XXX need review for Python 3. XXX need review for Windows Vista/Seven? How do I run a Python program under Windows? -------------------------------------------- This is not necessarily a straightforward question. If you are already familiar with running programs from the Windows command line then everything will seem obvious; otherwise, you might need a little more guidance. .. sidebar:: |Python Development on XP|_ :subtitle: `Python Development on XP`_ This series of screencasts aims to get you up and running with Python on Windows XP. The knowledge is distilled into 1.5 hours and will get you up and running with the right Python distribution, coding in your choice of IDE, and debugging and writing solid code with unit-tests. .. |Python Development on XP| image:: python-video-icon.png .. _`Python Development on XP`: http://showmedo.com/videotutorials/series?name=pythonOzsvaldPyNewbieSeries Unless you use some sort of integrated development environment, you will end up *typing* Windows commands into what is variously referred to as a "DOS window" or "Command prompt window". Usually you can create such a window from your Start menu; under Windows 7 the menu selection is :menuselection:`Start --> Programs --> Accessories --> Command Prompt`. You should be able to recognize when you have started such a window because you will see a Windows "command prompt", which usually looks like this: .. code-block:: doscon C:\> The letter may be different, and there might be other things after it, so you might just as easily see something like: .. code-block:: doscon D:\YourName\Projects\Python> depending on how your computer has been set up and what else you have recently done with it. Once you have started such a window, you are well on the way to running Python programs. You need to realize that your Python scripts have to be processed by another program called the Python *interpreter*. The interpreter reads your script, compiles it into bytecodes, and then executes the bytecodes to run your program. So, how do you arrange for the interpreter to handle your Python? First, you need to make sure that your command window recognises the word "python" as an instruction to start the interpreter. If you have opened a command window, you should try entering the command ``python`` and hitting return: .. code-block:: doscon C:\Users\YourName> python You should then see something like: .. code-block:: pycon Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> You have started the interpreter in "interactive mode". That means you can enter Python statements or expressions interactively and have them executed or evaluated while you wait. This is one of Python's strongest features. Check it by entering a few expressions of your choice and seeing the results: .. code-block:: pycon >>> print("Hello") Hello >>> "Hello" * 3 'HelloHelloHello' Many people use the interactive mode as a convenient yet highly programmable calculator. When you want to end your interactive Python session, hold the :kbd:`Ctrl` key down while you enter a :kbd:`Z`, then hit the ":kbd:`Enter`" key to get back to your Windows command prompt. You may also find that you have a Start-menu entry such as :menuselection:`Start --> Programs --> Python 3.3 --> Python (command line)` that results in you seeing the ``>>>`` prompt in a new window. If so, the window will disappear after you enter the :kbd:`Ctrl-Z` character; Windows is running a single "python" command in the window, and closes it when you terminate the interpreter. If the ``python`` command, instead of displaying the interpreter prompt ``>>>``, gives you a message like:: 'python' is not recognized as an internal or external command, operable program or batch file. .. sidebar:: |Adding Python to DOS Path|_ :subtitle: `Adding Python to DOS Path`_ Python is not added to the DOS path by default. This screencast will walk you through the steps to add the correct entry to the `System Path`, allowing Python to be executed from the command-line by all users. .. |Adding Python to DOS Path| image:: python-video-icon.png .. _`Adding Python to DOS Path`: http://showmedo.com/videotutorials/video?name=960000&fromSeriesID=96 or:: Bad command or filename then you need to make sure that your computer knows where to find the Python interpreter. To do this you will have to modify a setting called PATH, which is a list of directories where Windows will look for programs. You should arrange for Python's installation directory to be added to the PATH of every command window as it starts. If you installed Python fairly recently then the command :: dir C:\py* will probably tell you where it is installed; the usual location is something like ``C:\Python33``. Otherwise you will be reduced to a search of your whole disk ... use :menuselection:`Tools --> Find` or hit the :guilabel:`Search` button and look for "python.exe". Supposing you discover that Python is installed in the ``C:\Python33`` directory (the default at the time of writing), you should make sure that entering the command :: c:\Python33\python starts up the interpreter as above (and don't forget you'll need a ":kbd:`Ctrl-Z`" and an ":kbd:`Enter`" to get out of it). Once you have verified the directory, you can add it to the system path to make it easier to start Python by just running the ``python`` command. This is currently an option in the installer as of CPython 3.3. More information about environment variables can be found on the :ref:`Using Python on Windows <setting-envvars>` page. How do I make Python scripts executable? ---------------------------------------- On Windows, the standard Python installer already associates the .py extension with a file type (Python.File) and gives that file type an open command that runs the interpreter (``D:\Program Files\Python\python.exe "%1" %*``). This is enough to make scripts executable from the command prompt as 'foo.py'. If you'd rather be able to execute the script by simple typing 'foo' with no extension you need to add .py to the PATHEXT environment variable. Why does Python sometimes take so long to start? ------------------------------------------------ Usually Python starts very quickly on Windows, but occasionally there are bug reports that Python suddenly begins to take a long time to start up. This is made even more puzzling because Python will work fine on other Windows systems which appear to be configured identically. The problem may be caused by a misconfiguration of virus checking software on the problem machine. Some virus scanners have been known to introduce startup overhead of two orders of magnitude when the scanner is configured to monitor all reads from the filesystem. Try checking the configuration of virus scanning software on your systems to ensure that they are indeed configured identically. McAfee, when configured to scan all file system read activity, is a particular offender. How do I make an executable from a Python script? ------------------------------------------------- See http://cx-freeze.sourceforge.net/ for a distutils extension that allows you to create console and GUI executables from Python code. `py2exe <http://www.py2exe.org/>`_, the most popular extension for building Python 2.x-based executables, does not yet support Python 3 but a version that does is in development. Is a ``*.pyd`` file the same as a DLL? -------------------------------------- Yes, .pyd files are dll's, but there are a few differences. If you have a DLL named ``foo.pyd``, then it must have a function ``PyInit_foo()``. You can then write Python "import foo", and Python will search for foo.pyd (as well as foo.py, foo.pyc) and if it finds it, will attempt to call ``PyInit_foo()`` to initialize it. You do not link your .exe with foo.lib, as that would cause Windows to require the DLL to be present. Note that the search path for foo.pyd is PYTHONPATH, not the same as the path that Windows uses to search for foo.dll. Also, foo.pyd need not be present to run your program, whereas if you linked your program with a dll, the dll is required. Of course, foo.pyd is required if you want to say ``import foo``. In a DLL, linkage is declared in the source code with ``__declspec(dllexport)``. In a .pyd, linkage is defined in a list of available functions. How can I embed Python into a Windows application? -------------------------------------------------- Embedding the Python interpreter in a Windows app can be summarized as follows: 1. Do _not_ build Python into your .exe file directly. On Windows, Python must be a DLL to handle importing modules that are themselves DLL's. (This is the first key undocumented fact.) Instead, link to :file:`python{NN}.dll`; it is typically installed in ``C:\Windows\System``. *NN* is the Python version, a number such as "33" for Python 3.3. You can link to Python in two different ways. Load-time linking means linking against :file:`python{NN}.lib`, while run-time linking means linking against :file:`python{NN}.dll`. (General note: :file:`python{NN}.lib` is the so-called "import lib" corresponding to :file:`python{NN}.dll`. It merely defines symbols for the linker.) Run-time linking greatly simplifies link options; everything happens at run time. Your code must load :file:`python{NN}.dll` using the Windows ``LoadLibraryEx()`` routine. The code must also use access routines and data in :file:`python{NN}.dll` (that is, Python's C API's) using pointers obtained by the Windows ``GetProcAddress()`` routine. Macros can make using these pointers transparent to any C code that calls routines in Python's C API. Borland note: convert :file:`python{NN}.lib` to OMF format using Coff2Omf.exe first. .. XXX what about static linking? 2. If you use SWIG, it is easy to create a Python "extension module" that will make the app's data and methods available to Python. SWIG will handle just about all the grungy details for you. The result is C code that you link *into* your .exe file (!) You do _not_ have to create a DLL file, and this also simplifies linking. 3. SWIG will create an init function (a C function) whose name depends on the name of the extension module. For example, if the name of the module is leo, the init function will be called initleo(). If you use SWIG shadow classes, as you should, the init function will be called initleoc(). This initializes a mostly hidden helper class used by the shadow class. The reason you can link the C code in step 2 into your .exe file is that calling the initialization function is equivalent to importing the module into Python! (This is the second key undocumented fact.) 4. In short, you can use the following code to initialize the Python interpreter with your extension module. .. code-block:: c #include "python.h" ... Py_Initialize(); // Initialize Python. initmyAppc(); // Initialize (import) the helper class. PyRun_SimpleString("import myApp"); // Import the shadow class. 5. There are two problems with Python's C API which will become apparent if you use a compiler other than MSVC, the compiler used to build pythonNN.dll. Problem 1: The so-called "Very High Level" functions that take FILE * arguments will not work in a multi-compiler environment because each compiler's notion of a struct FILE will be different. From an implementation standpoint these are very _low_ level functions. Problem 2: SWIG generates the following code when generating wrappers to void functions: .. code-block:: c Py_INCREF(Py_None); _resultobj = Py_None; return _resultobj; Alas, Py_None is a macro that expands to a reference to a complex data structure called _Py_NoneStruct inside pythonNN.dll. Again, this code will fail in a mult-compiler environment. Replace such code by: .. code-block:: c return Py_BuildValue(""); It may be possible to use SWIG's ``%typemap`` command to make the change automatically, though I have not been able to get this to work (I'm a complete SWIG newbie). 6. Using a Python shell script to put up a Python interpreter window from inside your Windows app is not a good idea; the resulting window will be independent of your app's windowing system. Rather, you (or the wxPythonWindow class) should create a "native" interpreter window. It is easy to connect that window to the Python interpreter. You can redirect Python's i/o to _any_ object that supports read and write, so all you need is a Python object (defined in your extension module) that contains read() and write() methods. How do I keep editors from inserting tabs into my Python source? ---------------------------------------------------------------- The FAQ does not recommend using tabs, and the Python style guide, :pep:`8`, recommends 4 spaces for distributed Python code; this is also the Emacs python-mode default. Under any editor, mixing tabs and spaces is a bad idea. MSVC is no different in this respect, and is easily configured to use spaces: Take :menuselection:`Tools --> Options --> Tabs`, and for file type "Default" set "Tab size" and "Indent size" to 4, and select the "Insert spaces" radio button. Python raises :exc:`IndentationError` or :exc:`TabError` if mixed tabs and spaces are causing problems in leading whitespace. You may also run the :mod:`tabnanny` module to check a directory tree in batch mode. How do I check for a keypress without blocking? ----------------------------------------------- Use the msvcrt module. This is a standard Windows-specific extension module. It defines a function ``kbhit()`` which checks whether a keyboard hit is present, and ``getch()`` which gets one character without echoing it. How do I emulate os.kill() in Windows? -------------------------------------- Prior to Python 2.7 and 3.2, to terminate a process, you can use :mod:`ctypes`: .. code-block:: python import ctypes def kill(pid): """kill function for Win32""" kernel32 = ctypes.windll.kernel32 handle = kernel32.OpenProcess(1, 0, pid) return (0 != kernel32.TerminateProcess(handle, 0)) In 2.7 and 3.2, :func:`os.kill` is implemented similar to the above function, with the additional feature of being able to send :kbd:`Ctrl+C` and :kbd:`Ctrl+Break` to console subprocesses which are designed to handle those signals. See :func:`os.kill` for further details. How do I extract the downloaded documentation on Windows? --------------------------------------------------------- Sometimes, when you download the documentation package to a Windows machine using a web browser, the file extension of the saved file ends up being .EXE. This is a mistake; the extension should be .TGZ. Simply rename the downloaded file to have the .TGZ extension, and WinZip will be able to handle it. (If your copy of WinZip doesn't, get a newer one from https://www.winzip.com.) PK����� 3]#O\=��\=����faq/extending.rst.txtnu�[��������======================= Extending/Embedding FAQ ======================= .. only:: html .. contents:: .. highlight:: c .. XXX need review for Python 3. Can I create my own functions in C? ----------------------------------- Yes, you can create built-in modules containing functions, variables, exceptions and even new types in C. This is explained in the document :ref:`extending-index`. Most intermediate or advanced Python books will also cover this topic. Can I create my own functions in C++? ------------------------------------- Yes, using the C compatibility features found in C++. Place ``extern "C" { ... }`` around the Python include files and put ``extern "C"`` before each function that is going to be called by the Python interpreter. Global or static C++ objects with constructors are probably not a good idea. .. _c-wrapper-software: Writing C is hard; are there any alternatives? ---------------------------------------------- There are a number of alternatives to writing your own C extensions, depending on what you're trying to do. .. XXX make sure these all work `Cython <http://cython.org>`_ and its relative `Pyrex <https://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/>`_ are compilers that accept a slightly modified form of Python and generate the corresponding C code. Cython and Pyrex make it possible to write an extension without having to learn Python's C API. If you need to interface to some C or C++ library for which no Python extension currently exists, you can try wrapping the library's data types and functions with a tool such as `SWIG <http://www.swig.org>`_. `SIP <https://riverbankcomputing.com/software/sip/intro>`__, `CXX <http://cxx.sourceforge.net/>`_ `Boost <http://www.boost.org/libs/python/doc/index.html>`_, or `Weave <https://github.com/scipy/weave>`_ are also alternatives for wrapping C++ libraries. How can I execute arbitrary Python statements from C? ----------------------------------------------------- The highest-level function to do this is :c:func:`PyRun_SimpleString` which takes a single string argument to be executed in the context of the module ``__main__`` and returns ``0`` for success and ``-1`` when an exception occurred (including ``SyntaxError``). If you want more control, use :c:func:`PyRun_String`; see the source for :c:func:`PyRun_SimpleString` in ``Python/pythonrun.c``. How can I evaluate an arbitrary Python expression from C? --------------------------------------------------------- Call the function :c:func:`PyRun_String` from the previous question with the start symbol :c:data:`Py_eval_input`; it parses an expression, evaluates it and returns its value. How do I extract C values from a Python object? ----------------------------------------------- That depends on the object's type. If it's a tuple, :c:func:`PyTuple_Size` returns its length and :c:func:`PyTuple_GetItem` returns the item at a specified index. Lists have similar functions, :c:func:`PyListSize` and :c:func:`PyList_GetItem`. For bytes, :c:func:`PyBytes_Size` returns its length and :c:func:`PyBytes_AsStringAndSize` provides a pointer to its value and its length. Note that Python bytes objects may contain null bytes so C's :c:func:`strlen` should not be used. To test the type of an object, first make sure it isn't *NULL*, and then use :c:func:`PyBytes_Check`, :c:func:`PyTuple_Check`, :c:func:`PyList_Check`, etc. There is also a high-level API to Python objects which is provided by the so-called 'abstract' interface -- read ``Include/abstract.h`` for further details. It allows interfacing with any kind of Python sequence using calls like :c:func:`PySequence_Length`, :c:func:`PySequence_GetItem`, etc. as well as many other useful protocols such as numbers (:c:func:`PyNumber_Index` et al.) and mappings in the PyMapping APIs. How do I use Py_BuildValue() to create a tuple of arbitrary length? ------------------------------------------------------------------- You can't. Use :c:func:`PyTuple_Pack` instead. How do I call an object's method from C? ---------------------------------------- The :c:func:`PyObject_CallMethod` function can be used to call an arbitrary method of an object. The parameters are the object, the name of the method to call, a format string like that used with :c:func:`Py_BuildValue`, and the argument values:: PyObject * PyObject_CallMethod(PyObject *object, const char *method_name, const char *arg_format, ...); This works for any object that has methods -- whether built-in or user-defined. You are responsible for eventually :c:func:`Py_DECREF`\ 'ing the return value. To call, e.g., a file object's "seek" method with arguments 10, 0 (assuming the file object pointer is "f"):: res = PyObject_CallMethod(f, "seek", "(ii)", 10, 0); if (res == NULL) { ... an exception occurred ... } else { Py_DECREF(res); } Note that since :c:func:`PyObject_CallObject` *always* wants a tuple for the argument list, to call a function without arguments, pass "()" for the format, and to call a function with one argument, surround the argument in parentheses, e.g. "(i)". How do I catch the output from PyErr_Print() (or anything that prints to stdout/stderr)? ---------------------------------------------------------------------------------------- In Python code, define an object that supports the ``write()`` method. Assign this object to :data:`sys.stdout` and :data:`sys.stderr`. Call print_error, or just allow the standard traceback mechanism to work. Then, the output will go wherever your ``write()`` method sends it. The easiest way to do this is to use the :class:`io.StringIO` class: .. code-block:: pycon >>> import io, sys >>> sys.stdout = io.StringIO() >>> print('foo') >>> print('hello world!') >>> sys.stderr.write(sys.stdout.getvalue()) foo hello world! A custom object to do the same would look like this: .. code-block:: pycon >>> import io, sys >>> class StdoutCatcher(io.TextIOBase): ... def __init__(self): ... self.data = [] ... def write(self, stuff): ... self.data.append(stuff) ... >>> import sys >>> sys.stdout = StdoutCatcher() >>> print('foo') >>> print('hello world!') >>> sys.stderr.write(''.join(sys.stdout.data)) foo hello world! How do I access a module written in Python from C? -------------------------------------------------- You can get a pointer to the module object as follows:: module = PyImport_ImportModule("<modulename>"); If the module hasn't been imported yet (i.e. it is not yet present in :data:`sys.modules`), this initializes the module; otherwise it simply returns the value of ``sys.modules["<modulename>"]``. Note that it doesn't enter the module into any namespace -- it only ensures it has been initialized and is stored in :data:`sys.modules`. You can then access the module's attributes (i.e. any name defined in the module) as follows:: attr = PyObject_GetAttrString(module, "<attrname>"); Calling :c:func:`PyObject_SetAttrString` to assign to variables in the module also works. How do I interface to C++ objects from Python? ---------------------------------------------- Depending on your requirements, there are many approaches. To do this manually, begin by reading :ref:`the "Extending and Embedding" document <extending-index>`. Realize that for the Python run-time system, there isn't a whole lot of difference between C and C++ -- so the strategy of building a new Python type around a C structure (pointer) type will also work for C++ objects. For C++ libraries, see :ref:`c-wrapper-software`. I added a module using the Setup file and the make fails; why? -------------------------------------------------------------- Setup must end in a newline, if there is no newline there, the build process fails. (Fixing this requires some ugly shell script hackery, and this bug is so minor that it doesn't seem worth the effort.) How do I debug an extension? ---------------------------- When using GDB with dynamically loaded extensions, you can't set a breakpoint in your extension until your extension is loaded. In your ``.gdbinit`` file (or interactively), add the command: .. code-block:: none br _PyImport_LoadDynamicModule Then, when you run GDB: .. code-block:: shell-session $ gdb /local/bin/python gdb) run myscript.py gdb) continue # repeat until your extension is loaded gdb) finish # so that your extension is loaded gdb) br myfunction.c:50 gdb) continue I want to compile a Python module on my Linux system, but some files are missing. Why? -------------------------------------------------------------------------------------- Most packaged versions of Python don't include the :file:`/usr/lib/python2.{x}/config/` directory, which contains various files required for compiling Python extensions. For Red Hat, install the python-devel RPM to get the necessary files. For Debian, run ``apt-get install python-dev``. How do I tell "incomplete input" from "invalid input"? ------------------------------------------------------ Sometimes you want to emulate the Python interactive interpreter's behavior, where it gives you a continuation prompt when the input is incomplete (e.g. you typed the start of an "if" statement or you didn't close your parentheses or triple string quotes), but it gives you a syntax error message immediately when the input is invalid. In Python you can use the :mod:`codeop` module, which approximates the parser's behavior sufficiently. IDLE uses this, for example. The easiest way to do it in C is to call :c:func:`PyRun_InteractiveLoop` (perhaps in a separate thread) and let the Python interpreter handle the input for you. You can also set the :c:func:`PyOS_ReadlineFunctionPointer` to point at your custom input function. See ``Modules/readline.c`` and ``Parser/myreadline.c`` for more hints. However sometimes you have to run the embedded Python interpreter in the same thread as your rest application and you can't allow the :c:func:`PyRun_InteractiveLoop` to stop while waiting for user input. The one solution then is to call :c:func:`PyParser_ParseString` and test for ``e.error`` equal to ``E_EOF``, which means the input is incomplete). Here's a sample code fragment, untested, inspired by code from Alex Farber:: #include <Python.h> #include <node.h> #include <errcode.h> #include <grammar.h> #include <parsetok.h> #include <compile.h> int testcomplete(char *code) /* code should end in \n */ /* return -1 for error, 0 for incomplete, 1 for complete */ { node *n; perrdetail e; n = PyParser_ParseString(code, &_PyParser_Grammar, Py_file_input, &e); if (n == NULL) { if (e.error == E_EOF) return 0; return -1; } PyNode_Free(n); return 1; } Another solution is trying to compile the received string with :c:func:`Py_CompileString`. If it compiles without errors, try to execute the returned code object by calling :c:func:`PyEval_EvalCode`. Otherwise save the input for later. If the compilation fails, find out if it's an error or just more input is required - by extracting the message string from the exception tuple and comparing it to the string "unexpected EOF while parsing". Here is a complete example using the GNU readline library (you may want to ignore **SIGINT** while calling readline()):: #include <stdio.h> #include <readline.h> #include <Python.h> #include <object.h> #include <compile.h> #include <eval.h> int main (int argc, char* argv[]) { int i, j, done = 0; /* lengths of line, code */ char ps1[] = ">>> "; char ps2[] = "... "; char *prompt = ps1; char *msg, *line, *code = NULL; PyObject *src, *glb, *loc; PyObject *exc, *val, *trb, *obj, *dum; Py_Initialize (); loc = PyDict_New (); glb = PyDict_New (); PyDict_SetItemString (glb, "__builtins__", PyEval_GetBuiltins ()); while (!done) { line = readline (prompt); if (NULL == line) /* Ctrl-D pressed */ { done = 1; } else { i = strlen (line); if (i > 0) add_history (line); /* save non-empty lines */ if (NULL == code) /* nothing in code yet */ j = 0; else j = strlen (code); code = realloc (code, i + j + 2); if (NULL == code) /* out of memory */ exit (1); if (0 == j) /* code was empty, so */ code[0] = '\0'; /* keep strncat happy */ strncat (code, line, i); /* append line to code */ code[i + j] = '\n'; /* append '\n' to code */ code[i + j + 1] = '\0'; src = Py_CompileString (code, "<stdin>", Py_single_input); if (NULL != src) /* compiled just fine - */ { if (ps1 == prompt || /* ">>> " or */ '\n' == code[i + j - 1]) /* "... " and double '\n' */ { /* so execute it */ dum = PyEval_EvalCode (src, glb, loc); Py_XDECREF (dum); Py_XDECREF (src); free (code); code = NULL; if (PyErr_Occurred ()) PyErr_Print (); prompt = ps1; } } /* syntax error or E_EOF? */ else if (PyErr_ExceptionMatches (PyExc_SyntaxError)) { PyErr_Fetch (&exc, &val, &trb); /* clears exception! */ if (PyArg_ParseTuple (val, "sO", &msg, &obj) && !strcmp (msg, "unexpected EOF while parsing")) /* E_EOF */ { Py_XDECREF (exc); Py_XDECREF (val); Py_XDECREF (trb); prompt = ps2; } else /* some other syntax error */ { PyErr_Restore (exc, val, trb); PyErr_Print (); free (code); code = NULL; prompt = ps1; } } else /* some non-syntax error */ { PyErr_Print (); free (code); code = NULL; prompt = ps1; } free (line); } } Py_XDECREF(glb); Py_XDECREF(loc); Py_Finalize(); exit(0); } How do I find undefined g++ symbols __builtin_new or __pure_virtual? -------------------------------------------------------------------- To dynamically load g++ extension modules, you must recompile Python, relink it using g++ (change LINKCC in the Python Modules Makefile), and link your extension module using g++ (e.g., ``g++ -shared -o mymodule.so mymodule.o``). Can I create an object class with some methods implemented in C and others in Python (e.g. through inheritance)? ---------------------------------------------------------------------------------------------------------------- Yes, you can inherit from built-in classes such as :class:`int`, :class:`list`, :class:`dict`, etc. The Boost Python Library (BPL, http://www.boost.org/libs/python/doc/index.html) provides a way of doing this from C++ (i.e. you can inherit from an extension class written in C++ using the BPL). PK����� 3] Xa�a���faq/programming.rst.txtnu�[��������:tocdepth: 2 =============== Programming FAQ =============== .. only:: html .. contents:: General Questions ================= Is there a source code level debugger with breakpoints, single-stepping, etc.? ------------------------------------------------------------------------------ Yes. The pdb module is a simple but adequate console-mode debugger for Python. It is part of the standard Python library, and is :mod:`documented in the Library Reference Manual <pdb>`. You can also write your own debugger by using the code for pdb as an example. The IDLE interactive development environment, which is part of the standard Python distribution (normally available as Tools/scripts/idle), includes a graphical debugger. PythonWin is a Python IDE that includes a GUI debugger based on pdb. The Pythonwin debugger colors breakpoints and has quite a few cool features such as debugging non-Pythonwin programs. Pythonwin is available as part of the `Python for Windows Extensions <https://sourceforge.net/projects/pywin32/>`__ project and as a part of the ActivePython distribution (see https://www.activestate.com/activepython\ ). `Boa Constructor <http://boa-constructor.sourceforge.net/>`_ is an IDE and GUI builder that uses wxWidgets. It offers visual frame creation and manipulation, an object inspector, many views on the source like object browsers, inheritance hierarchies, doc string generated html documentation, an advanced debugger, integrated help, and Zope support. `Eric <http://eric-ide.python-projects.org/>`_ is an IDE built on PyQt and the Scintilla editing component. Pydb is a version of the standard Python debugger pdb, modified for use with DDD (Data Display Debugger), a popular graphical debugger front end. Pydb can be found at http://bashdb.sourceforge.net/pydb/ and DDD can be found at https://www.gnu.org/software/ddd. There are a number of commercial Python IDEs that include graphical debuggers. They include: * Wing IDE (https://wingware.com/) * Komodo IDE (https://komodoide.com/) * PyCharm (https://www.jetbrains.com/pycharm/) Is there a tool to help find bugs or perform static analysis? ------------------------------------------------------------- Yes. PyChecker is a static analysis tool that finds bugs in Python source code and warns about code complexity and style. You can get PyChecker from http://pychecker.sourceforge.net/. `Pylint <https://www.pylint.org/>`_ is another tool that checks if a module satisfies a coding standard, and also makes it possible to write plug-ins to add a custom feature. In addition to the bug checking that PyChecker performs, Pylint offers some additional features such as checking line length, whether variable names are well-formed according to your coding standard, whether declared interfaces are fully implemented, and more. https://docs.pylint.org/ provides a full list of Pylint's features. How can I create a stand-alone binary from a Python script? ----------------------------------------------------------- You don't need the ability to compile Python to C code if all you want is a stand-alone program that users can download and run without having to install the Python distribution first. There are a number of tools that determine the set of modules required by a program and bind these modules together with a Python binary to produce a single executable. One is to use the freeze tool, which is included in the Python source tree as ``Tools/freeze``. It converts Python byte code to C arrays; a C compiler you can embed all your modules into a new program, which is then linked with the standard Python modules. It works by scanning your source recursively for import statements (in both forms) and looking for the modules in the standard Python path as well as in the source directory (for built-in modules). It then turns the bytecode for modules written in Python into C code (array initializers that can be turned into code objects using the marshal module) and creates a custom-made config file that only contains those built-in modules which are actually used in the program. It then compiles the generated C code and links it with the rest of the Python interpreter to form a self-contained binary which acts exactly like your script. Obviously, freeze requires a C compiler. There are several other utilities which don't. One is Thomas Heller's py2exe (Windows only) at http://www.py2exe.org/ Another tool is Anthony Tuininga's `cx_Freeze <http://cx-freeze.sourceforge.net/>`_. Are there coding standards or a style guide for Python programs? ---------------------------------------------------------------- Yes. The coding style required for standard library modules is documented as :pep:`8`. Core Language ============= Why am I getting an UnboundLocalError when the variable has a value? -------------------------------------------------------------------- It can be a surprise to get the UnboundLocalError in previously working code when it is modified by adding an assignment statement somewhere in the body of a function. This code: >>> x = 10 >>> def bar(): ... print(x) >>> bar() 10 works, but this code: >>> x = 10 >>> def foo(): ... print(x) ... x += 1 results in an UnboundLocalError: >>> foo() Traceback (most recent call last): ... UnboundLocalError: local variable 'x' referenced before assignment This is because when you make an assignment to a variable in a scope, that variable becomes local to that scope and shadows any similarly named variable in the outer scope. Since the last statement in foo assigns a new value to ``x``, the compiler recognizes it as a local variable. Consequently when the earlier ``print(x)`` attempts to print the uninitialized local variable and an error results. In the example above you can access the outer scope variable by declaring it global: >>> x = 10 >>> def foobar(): ... global x ... print(x) ... x += 1 >>> foobar() 10 This explicit declaration is required in order to remind you that (unlike the superficially analogous situation with class and instance variables) you are actually modifying the value of the variable in the outer scope: >>> print(x) 11 You can do a similar thing in a nested scope using the :keyword:`nonlocal` keyword: >>> def foo(): ... x = 10 ... def bar(): ... nonlocal x ... print(x) ... x += 1 ... bar() ... print(x) >>> foo() 10 11 What are the rules for local and global variables in Python? ------------------------------------------------------------ In Python, variables that are only referenced inside a function are implicitly global. If a variable is assigned a value anywhere within the function's body, it's assumed to be a local unless explicitly declared as global. Though a bit surprising at first, a moment's consideration explains this. On one hand, requiring :keyword:`global` for assigned variables provides a bar against unintended side-effects. On the other hand, if ``global`` was required for all global references, you'd be using ``global`` all the time. You'd have to declare as global every reference to a built-in function or to a component of an imported module. This clutter would defeat the usefulness of the ``global`` declaration for identifying side-effects. Why do lambdas defined in a loop with different values all return the same result? ---------------------------------------------------------------------------------- Assume you use a for loop to define a few different lambdas (or even plain functions), e.g.:: >>> squares = [] >>> for x in range(5): ... squares.append(lambda: x**2) This gives you a list that contains 5 lambdas that calculate ``x**2``. You might expect that, when called, they would return, respectively, ``0``, ``1``, ``4``, ``9``, and ``16``. However, when you actually try you will see that they all return ``16``:: >>> squares[2]() 16 >>> squares[4]() 16 This happens because ``x`` is not local to the lambdas, but is defined in the outer scope, and it is accessed when the lambda is called --- not when it is defined. At the end of the loop, the value of ``x`` is ``4``, so all the functions now return ``4**2``, i.e. ``16``. You can also verify this by changing the value of ``x`` and see how the results of the lambdas change:: >>> x = 8 >>> squares[2]() 64 In order to avoid this, you need to save the values in variables local to the lambdas, so that they don't rely on the value of the global ``x``:: >>> squares = [] >>> for x in range(5): ... squares.append(lambda n=x: n**2) Here, ``n=x`` creates a new variable ``n`` local to the lambda and computed when the lambda is defined so that it has the same value that ``x`` had at that point in the loop. This means that the value of ``n`` will be ``0`` in the first lambda, ``1`` in the second, ``2`` in the third, and so on. Therefore each lambda will now return the correct result:: >>> squares[2]() 4 >>> squares[4]() 16 Note that this behaviour is not peculiar to lambdas, but applies to regular functions too. How do I share global variables across modules? ------------------------------------------------ The canonical way to share information across modules within a single program is to create a special module (often called config or cfg). Just import the config module in all modules of your application; the module then becomes available as a global name. Because there is only one instance of each module, any changes made to the module object get reflected everywhere. For example: config.py:: x = 0 # Default value of the 'x' configuration setting mod.py:: import config config.x = 1 main.py:: import config import mod print(config.x) Note that using a module is also the basis for implementing the Singleton design pattern, for the same reason. What are the "best practices" for using import in a module? ----------------------------------------------------------- In general, don't use ``from modulename import *``. Doing so clutters the importer's namespace, and makes it much harder for linters to detect undefined names. Import modules at the top of a file. Doing so makes it clear what other modules your code requires and avoids questions of whether the module name is in scope. Using one import per line makes it easy to add and delete module imports, but using multiple imports per line uses less screen space. It's good practice if you import modules in the following order: 1. standard library modules -- e.g. ``sys``, ``os``, ``getopt``, ``re`` 2. third-party library modules (anything installed in Python's site-packages directory) -- e.g. mx.DateTime, ZODB, PIL.Image, etc. 3. locally-developed modules It is sometimes necessary to move imports to a function or class to avoid problems with circular imports. Gordon McMillan says: Circular imports are fine where both modules use the "import <module>" form of import. They fail when the 2nd module wants to grab a name out of the first ("from module import name") and the import is at the top level. That's because names in the 1st are not yet available, because the first module is busy importing the 2nd. In this case, if the second module is only used in one function, then the import can easily be moved into that function. By the time the import is called, the first module will have finished initializing, and the second module can do its import. It may also be necessary to move imports out of the top level of code if some of the modules are platform-specific. In that case, it may not even be possible to import all of the modules at the top of the file. In this case, importing the correct modules in the corresponding platform-specific code is a good option. Only move imports into a local scope, such as inside a function definition, if it's necessary to solve a problem such as avoiding a circular import or are trying to reduce the initialization time of a module. This technique is especially helpful if many of the imports are unnecessary depending on how the program executes. You may also want to move imports into a function if the modules are only ever used in that function. Note that loading a module the first time may be expensive because of the one time initialization of the module, but loading a module multiple times is virtually free, costing only a couple of dictionary lookups. Even if the module name has gone out of scope, the module is probably available in :data:`sys.modules`. Why are default values shared between objects? ---------------------------------------------- This type of bug commonly bites neophyte programmers. Consider this function:: def foo(mydict={}): # Danger: shared reference to one dict for all calls ... compute something ... mydict[key] = value return mydict The first time you call this function, ``mydict`` contains a single item. The second time, ``mydict`` contains two items because when ``foo()`` begins executing, ``mydict`` starts out with an item already in it. It is often expected that a function call creates new objects for default values. This is not what happens. Default values are created exactly once, when the function is defined. If that object is changed, like the dictionary in this example, subsequent calls to the function will refer to this changed object. By definition, immutable objects such as numbers, strings, tuples, and ``None``, are safe from change. Changes to mutable objects such as dictionaries, lists, and class instances can lead to confusion. Because of this feature, it is good programming practice to not use mutable objects as default values. Instead, use ``None`` as the default value and inside the function, check if the parameter is ``None`` and create a new list/dictionary/whatever if it is. For example, don't write:: def foo(mydict={}): ... but:: def foo(mydict=None): if mydict is None: mydict = {} # create a new dict for local namespace This feature can be useful. When you have a function that's time-consuming to compute, a common technique is to cache the parameters and the resulting value of each call to the function, and return the cached value if the same value is requested again. This is called "memoizing", and can be implemented like this:: # Callers can only provide two parameters and optionally pass _cache by keyword def expensive(arg1, arg2, *, _cache={}): if (arg1, arg2) in _cache: return _cache[(arg1, arg2)] # Calculate the value result = ... expensive computation ... _cache[(arg1, arg2)] = result # Store result in the cache return result You could use a global variable containing a dictionary instead of the default value; it's a matter of taste. How can I pass optional or keyword parameters from one function to another? --------------------------------------------------------------------------- Collect the arguments using the ``*`` and ``**`` specifiers in the function's parameter list; this gives you the positional arguments as a tuple and the keyword arguments as a dictionary. You can then pass these arguments when calling another function by using ``*`` and ``**``:: def f(x, *args, **kwargs): ... kwargs['width'] = '14.3c' ... g(x, *args, **kwargs) .. index:: single: argument; difference from parameter single: parameter; difference from argument .. _faq-argument-vs-parameter: What is the difference between arguments and parameters? -------------------------------------------------------- :term:`Parameters <parameter>` are defined by the names that appear in a function definition, whereas :term:`arguments <argument>` are the values actually passed to a function when calling it. Parameters define what types of arguments a function can accept. For example, given the function definition:: def func(foo, bar=None, **kwargs): pass *foo*, *bar* and *kwargs* are parameters of ``func``. However, when calling ``func``, for example:: func(42, bar=314, extra=somevar) the values ``42``, ``314``, and ``somevar`` are arguments. Why did changing list 'y' also change list 'x'? ------------------------------------------------ If you wrote code like:: >>> x = [] >>> y = x >>> y.append(10) >>> y [10] >>> x [10] you might be wondering why appending an element to ``y`` changed ``x`` too. There are two factors that produce this result: 1) Variables are simply names that refer to objects. Doing ``y = x`` doesn't create a copy of the list -- it creates a new variable ``y`` that refers to the same object ``x`` refers to. This means that there is only one object (the list), and both ``x`` and ``y`` refer to it. 2) Lists are :term:`mutable`, which means that you can change their content. After the call to :meth:`~list.append`, the content of the mutable object has changed from ``[]`` to ``[10]``. Since both the variables refer to the same object, using either name accesses the modified value ``[10]``. If we instead assign an immutable object to ``x``:: >>> x = 5 # ints are immutable >>> y = x >>> x = x + 1 # 5 can't be mutated, we are creating a new object here >>> x 6 >>> y 5 we can see that in this case ``x`` and ``y`` are not equal anymore. This is because integers are :term:`immutable`, and when we do ``x = x + 1`` we are not mutating the int ``5`` by incrementing its value; instead, we are creating a new object (the int ``6``) and assigning it to ``x`` (that is, changing which object ``x`` refers to). After this assignment we have two objects (the ints ``6`` and ``5``) and two variables that refer to them (``x`` now refers to ``6`` but ``y`` still refers to ``5``). Some operations (for example ``y.append(10)`` and ``y.sort()``) mutate the object, whereas superficially similar operations (for example ``y = y + [10]`` and ``sorted(y)``) create a new object. In general in Python (and in all cases in the standard library) a method that mutates an object will return ``None`` to help avoid getting the two types of operations confused. So if you mistakenly write ``y.sort()`` thinking it will give you a sorted copy of ``y``, you'll instead end up with ``None``, which will likely cause your program to generate an easily diagnosed error. However, there is one class of operations where the same operation sometimes has different behaviors with different types: the augmented assignment operators. For example, ``+=`` mutates lists but not tuples or ints (``a_list += [1, 2, 3]`` is equivalent to ``a_list.extend([1, 2, 3])`` and mutates ``a_list``, whereas ``some_tuple += (1, 2, 3)`` and ``some_int += 1`` create new objects). In other words: * If we have a mutable object (:class:`list`, :class:`dict`, :class:`set`, etc.), we can use some specific operations to mutate it and all the variables that refer to it will see the change. * If we have an immutable object (:class:`str`, :class:`int`, :class:`tuple`, etc.), all the variables that refer to it will always see the same value, but operations that transform that value into a new value always return a new object. If you want to know if two variables refer to the same object or not, you can use the :keyword:`is` operator, or the built-in function :func:`id`. How do I write a function with output parameters (call by reference)? --------------------------------------------------------------------- Remember that arguments are passed by assignment in Python. Since assignment just creates references to objects, there's no alias between an argument name in the caller and callee, and so no call-by-reference per se. You can achieve the desired effect in a number of ways. 1) By returning a tuple of the results:: def func2(a, b): a = 'new-value' # a and b are local names b = b + 1 # assigned to new objects return a, b # return new values x, y = 'old-value', 99 x, y = func2(x, y) print(x, y) # output: new-value 100 This is almost always the clearest solution. 2) By using global variables. This isn't thread-safe, and is not recommended. 3) By passing a mutable (changeable in-place) object:: def func1(a): a[0] = 'new-value' # 'a' references a mutable list a[1] = a[1] + 1 # changes a shared object args = ['old-value', 99] func1(args) print(args[0], args[1]) # output: new-value 100 4) By passing in a dictionary that gets mutated:: def func3(args): args['a'] = 'new-value' # args is a mutable dictionary args['b'] = args['b'] + 1 # change it in-place args = {'a': 'old-value', 'b': 99} func3(args) print(args['a'], args['b']) 5) Or bundle up values in a class instance:: class callByRef: def __init__(self, **args): for (key, value) in args.items(): setattr(self, key, value) def func4(args): args.a = 'new-value' # args is a mutable callByRef args.b = args.b + 1 # change object in-place args = callByRef(a='old-value', b=99) func4(args) print(args.a, args.b) There's almost never a good reason to get this complicated. Your best choice is to return a tuple containing the multiple results. How do you make a higher order function in Python? -------------------------------------------------- You have two choices: you can use nested scopes or you can use callable objects. For example, suppose you wanted to define ``linear(a,b)`` which returns a function ``f(x)`` that computes the value ``a*x+b``. Using nested scopes:: def linear(a, b): def result(x): return a * x + b return result Or using a callable object:: class linear: def __init__(self, a, b): self.a, self.b = a, b def __call__(self, x): return self.a * x + self.b In both cases, :: taxes = linear(0.3, 2) gives a callable object where ``taxes(10e6) == 0.3 * 10e6 + 2``. The callable object approach has the disadvantage that it is a bit slower and results in slightly longer code. However, note that a collection of callables can share their signature via inheritance:: class exponential(linear): # __init__ inherited def __call__(self, x): return self.a * (x ** self.b) Object can encapsulate state for several methods:: class counter: value = 0 def set(self, x): self.value = x def up(self): self.value = self.value + 1 def down(self): self.value = self.value - 1 count = counter() inc, dec, reset = count.up, count.down, count.set Here ``inc()``, ``dec()`` and ``reset()`` act like functions which share the same counting variable. How do I copy an object in Python? ---------------------------------- In general, try :func:`copy.copy` or :func:`copy.deepcopy` for the general case. Not all objects can be copied, but most can. Some objects can be copied more easily. Dictionaries have a :meth:`~dict.copy` method:: newdict = olddict.copy() Sequences can be copied by slicing:: new_l = l[:] How can I find the methods or attributes of an object? ------------------------------------------------------ For an instance x of a user-defined class, ``dir(x)`` returns an alphabetized list of the names containing the instance attributes and methods and attributes defined by its class. How can my code discover the name of an object? ----------------------------------------------- Generally speaking, it can't, because objects don't really have names. Essentially, assignment always binds a name to a value; The same is true of ``def`` and ``class`` statements, but in that case the value is a callable. Consider the following code:: >>> class A: ... pass ... >>> B = A >>> a = B() >>> b = a >>> print(b) <__main__.A object at 0x16D07CC> >>> print(a) <__main__.A object at 0x16D07CC> Arguably the class has a name: even though it is bound to two names and invoked through the name B the created instance is still reported as an instance of class A. However, it is impossible to say whether the instance's name is a or b, since both names are bound to the same value. Generally speaking it should not be necessary for your code to "know the names" of particular values. Unless you are deliberately writing introspective programs, this is usually an indication that a change of approach might be beneficial. In comp.lang.python, Fredrik Lundh once gave an excellent analogy in answer to this question: The same way as you get the name of that cat you found on your porch: the cat (object) itself cannot tell you its name, and it doesn't really care -- so the only way to find out what it's called is to ask all your neighbours (namespaces) if it's their cat (object)... ....and don't be surprised if you'll find that it's known by many names, or no name at all! What's up with the comma operator's precedence? ----------------------------------------------- Comma is not an operator in Python. Consider this session:: >>> "a" in "b", "a" (False, 'a') Since the comma is not an operator, but a separator between expressions the above is evaluated as if you had entered:: ("a" in "b"), "a" not:: "a" in ("b", "a") The same is true of the various assignment operators (``=``, ``+=`` etc). They are not truly operators but syntactic delimiters in assignment statements. Is there an equivalent of C's "?:" ternary operator? ---------------------------------------------------- Yes, there is. The syntax is as follows:: [on_true] if [expression] else [on_false] x, y = 50, 25 small = x if x < y else y Before this syntax was introduced in Python 2.5, a common idiom was to use logical operators:: [expression] and [on_true] or [on_false] However, this idiom is unsafe, as it can give wrong results when *on_true* has a false boolean value. Therefore, it is always better to use the ``... if ... else ...`` form. Is it possible to write obfuscated one-liners in Python? -------------------------------------------------------- Yes. Usually this is done by nesting :keyword:`lambda` within :keyword:`lambda`. See the following three examples, due to Ulf Bartelt:: from functools import reduce # Primes < 1000 print(list(filter(None,map(lambda y:y*reduce(lambda x,y:x*y!=0, map(lambda x,y=y:y%x,range(2,int(pow(y,0.5)+1))),1),range(2,1000))))) # First 10 Fibonacci numbers print(list(map(lambda x,f=lambda x,f:(f(x-1,f)+f(x-2,f)) if x>1 else 1: f(x,f), range(10)))) # Mandelbrot set print((lambda Ru,Ro,Iu,Io,IM,Sx,Sy:reduce(lambda x,y:x+y,map(lambda y, Iu=Iu,Io=Io,Ru=Ru,Ro=Ro,Sy=Sy,L=lambda yc,Iu=Iu,Io=Io,Ru=Ru,Ro=Ro,i=IM, Sx=Sx,Sy=Sy:reduce(lambda x,y:x+y,map(lambda x,xc=Ru,yc=yc,Ru=Ru,Ro=Ro, i=i,Sx=Sx,F=lambda xc,yc,x,y,k,f=lambda xc,yc,x,y,k,f:(k<=0)or (x*x+y*y >=4.0) or 1+f(xc,yc,x*x-y*y+xc,2.0*x*y+yc,k-1,f):f(xc,yc,x,y,k,f):chr( 64+F(Ru+x*(Ro-Ru)/Sx,yc,0,0,i)),range(Sx))):L(Iu+y*(Io-Iu)/Sy),range(Sy ))))(-2.1, 0.7, -1.2, 1.2, 30, 80, 24)) # \___ ___/ \___ ___/ | | |__ lines on screen # V V | |______ columns on screen # | | |__________ maximum of "iterations" # | |_________________ range on y axis # |____________________________ range on x axis Don't try this at home, kids! Numbers and strings =================== How do I specify hexadecimal and octal integers? ------------------------------------------------ To specify an octal digit, precede the octal value with a zero, and then a lower or uppercase "o". For example, to set the variable "a" to the octal value "10" (8 in decimal), type:: >>> a = 0o10 >>> a 8 Hexadecimal is just as easy. Simply precede the hexadecimal number with a zero, and then a lower or uppercase "x". Hexadecimal digits can be specified in lower or uppercase. For example, in the Python interpreter:: >>> a = 0xa5 >>> a 165 >>> b = 0XB2 >>> b 178 Why does -22 // 10 return -3? ----------------------------- It's primarily driven by the desire that ``i % j`` have the same sign as ``j``. If you want that, and also want:: i == (i // j) * j + (i % j) then integer division has to return the floor. C also requires that identity to hold, and then compilers that truncate ``i // j`` need to make ``i % j`` have the same sign as ``i``. There are few real use cases for ``i % j`` when ``j`` is negative. When ``j`` is positive, there are many, and in virtually all of them it's more useful for ``i % j`` to be ``>= 0``. If the clock says 10 now, what did it say 200 hours ago? ``-190 % 12 == 2`` is useful; ``-190 % 12 == -10`` is a bug waiting to bite. How do I convert a string to a number? -------------------------------------- For integers, use the built-in :func:`int` type constructor, e.g. ``int('144') == 144``. Similarly, :func:`float` converts to floating-point, e.g. ``float('144') == 144.0``. By default, these interpret the number as decimal, so that ``int('0144') == 144`` and ``int('0x144')`` raises :exc:`ValueError`. ``int(string, base)`` takes the base to convert from as a second optional argument, so ``int('0x144', 16) == 324``. If the base is specified as 0, the number is interpreted using Python's rules: a leading '0o' indicates octal, and '0x' indicates a hex number. Do not use the built-in function :func:`eval` if all you need is to convert strings to numbers. :func:`eval` will be significantly slower and it presents a security risk: someone could pass you a Python expression that might have unwanted side effects. For example, someone could pass ``__import__('os').system("rm -rf $HOME")`` which would erase your home directory. :func:`eval` also has the effect of interpreting numbers as Python expressions, so that e.g. ``eval('09')`` gives a syntax error because Python does not allow leading '0' in a decimal number (except '0'). How do I convert a number to a string? -------------------------------------- To convert, e.g., the number 144 to the string '144', use the built-in type constructor :func:`str`. If you want a hexadecimal or octal representation, use the built-in functions :func:`hex` or :func:`oct`. For fancy formatting, see the :ref:`f-strings` and :ref:`formatstrings` sections, e.g. ``"{:04d}".format(144)`` yields ``'0144'`` and ``"{:.3f}".format(1.0/3.0)`` yields ``'0.333'``. How do I modify a string in place? ---------------------------------- You can't, because strings are immutable. In most situations, you should simply construct a new string from the various parts you want to assemble it from. However, if you need an object with the ability to modify in-place unicode data, try using an :class:`io.StringIO` object or the :mod:`array` module:: >>> import io >>> s = "Hello, world" >>> sio = io.StringIO(s) >>> sio.getvalue() 'Hello, world' >>> sio.seek(7) 7 >>> sio.write("there!") 6 >>> sio.getvalue() 'Hello, there!' >>> import array >>> a = array.array('u', s) >>> print(a) array('u', 'Hello, world') >>> a[0] = 'y' >>> print(a) array('u', 'yello, world') >>> a.tounicode() 'yello, world' How do I use strings to call functions/methods? ----------------------------------------------- There are various techniques. * The best is to use a dictionary that maps strings to functions. The primary advantage of this technique is that the strings do not need to match the names of the functions. This is also the primary technique used to emulate a case construct:: def a(): pass def b(): pass dispatch = {'go': a, 'stop': b} # Note lack of parens for funcs dispatch[get_input()]() # Note trailing parens to call function * Use the built-in function :func:`getattr`:: import foo getattr(foo, 'bar')() Note that :func:`getattr` works on any object, including classes, class instances, modules, and so on. This is used in several places in the standard library, like this:: class Foo: def do_foo(self): ... def do_bar(self): ... f = getattr(foo_instance, 'do_' + opname) f() * Use :func:`locals` or :func:`eval` to resolve the function name:: def myFunc(): print("hello") fname = "myFunc" f = locals()[fname] f() f = eval(fname) f() Note: Using :func:`eval` is slow and dangerous. If you don't have absolute control over the contents of the string, someone could pass a string that resulted in an arbitrary function being executed. Is there an equivalent to Perl's chomp() for removing trailing newlines from strings? ------------------------------------------------------------------------------------- You can use ``S.rstrip("\r\n")`` to remove all occurrences of any line terminator from the end of the string ``S`` without removing other trailing whitespace. If the string ``S`` represents more than one line, with several empty lines at the end, the line terminators for all the blank lines will be removed:: >>> lines = ("line 1 \r\n" ... "\r\n" ... "\r\n") >>> lines.rstrip("\n\r") 'line 1 ' Since this is typically only desired when reading text one line at a time, using ``S.rstrip()`` this way works well. Is there a scanf() or sscanf() equivalent? ------------------------------------------ Not as such. For simple input parsing, the easiest approach is usually to split the line into whitespace-delimited words using the :meth:`~str.split` method of string objects and then convert decimal strings to numeric values using :func:`int` or :func:`float`. ``split()`` supports an optional "sep" parameter which is useful if the line uses something other than whitespace as a separator. For more complicated input parsing, regular expressions are more powerful than C's :c:func:`sscanf` and better suited for the task. What does 'UnicodeDecodeError' or 'UnicodeEncodeError' error mean? ------------------------------------------------------------------- See the :ref:`unicode-howto`. Performance =========== My program is too slow. How do I speed it up? --------------------------------------------- That's a tough one, in general. First, here are a list of things to remember before diving further: * Performance characteristics vary across Python implementations. This FAQ focusses on :term:`CPython`. * Behaviour can vary across operating systems, especially when talking about I/O or multi-threading. * You should always find the hot spots in your program *before* attempting to optimize any code (see the :mod:`profile` module). * Writing benchmark scripts will allow you to iterate quickly when searching for improvements (see the :mod:`timeit` module). * It is highly recommended to have good code coverage (through unit testing or any other technique) before potentially introducing regressions hidden in sophisticated optimizations. That being said, there are many tricks to speed up Python code. Here are some general principles which go a long way towards reaching acceptable performance levels: * Making your algorithms faster (or changing to faster ones) can yield much larger benefits than trying to sprinkle micro-optimization tricks all over your code. * Use the right data structures. Study documentation for the :ref:`bltin-types` and the :mod:`collections` module. * When the standard library provides a primitive for doing something, it is likely (although not guaranteed) to be faster than any alternative you may come up with. This is doubly true for primitives written in C, such as builtins and some extension types. For example, be sure to use either the :meth:`list.sort` built-in method or the related :func:`sorted` function to do sorting (and see the :ref:`sortinghowto` for examples of moderately advanced usage). * Abstractions tend to create indirections and force the interpreter to work more. If the levels of indirection outweigh the amount of useful work done, your program will be slower. You should avoid excessive abstraction, especially under the form of tiny functions or methods (which are also often detrimental to readability). If you have reached the limit of what pure Python can allow, there are tools to take you further away. For example, `Cython <http://cython.org>`_ can compile a slightly modified version of Python code into a C extension, and can be used on many different platforms. Cython can take advantage of compilation (and optional type annotations) to make your code significantly faster than when interpreted. If you are confident in your C programming skills, you can also :ref:`write a C extension module <extending-index>` yourself. .. seealso:: The wiki page devoted to `performance tips <https://wiki.python.org/moin/PythonSpeed/PerformanceTips>`_. .. _efficient_string_concatenation: What is the most efficient way to concatenate many strings together? -------------------------------------------------------------------- :class:`str` and :class:`bytes` objects are immutable, therefore concatenating many strings together is inefficient as each concatenation creates a new object. In the general case, the total runtime cost is quadratic in the total string length. To accumulate many :class:`str` objects, the recommended idiom is to place them into a list and call :meth:`str.join` at the end:: chunks = [] for s in my_strings: chunks.append(s) result = ''.join(chunks) (another reasonably efficient idiom is to use :class:`io.StringIO`) To accumulate many :class:`bytes` objects, the recommended idiom is to extend a :class:`bytearray` object using in-place concatenation (the ``+=`` operator):: result = bytearray() for b in my_bytes_objects: result += b Sequences (Tuples/Lists) ======================== How do I convert between tuples and lists? ------------------------------------------ The type constructor ``tuple(seq)`` converts any sequence (actually, any iterable) into a tuple with the same items in the same order. For example, ``tuple([1, 2, 3])`` yields ``(1, 2, 3)`` and ``tuple('abc')`` yields ``('a', 'b', 'c')``. If the argument is a tuple, it does not make a copy but returns the same object, so it is cheap to call :func:`tuple` when you aren't sure that an object is already a tuple. The type constructor ``list(seq)`` converts any sequence or iterable into a list with the same items in the same order. For example, ``list((1, 2, 3))`` yields ``[1, 2, 3]`` and ``list('abc')`` yields ``['a', 'b', 'c']``. If the argument is a list, it makes a copy just like ``seq[:]`` would. What's a negative index? ------------------------ Python sequences are indexed with positive numbers and negative numbers. For positive numbers 0 is the first index 1 is the second index and so forth. For negative indices -1 is the last index and -2 is the penultimate (next to last) index and so forth. Think of ``seq[-n]`` as the same as ``seq[len(seq)-n]``. Using negative indices can be very convenient. For example ``S[:-1]`` is all of the string except for its last character, which is useful for removing the trailing newline from a string. How do I iterate over a sequence in reverse order? -------------------------------------------------- Use the :func:`reversed` built-in function, which is new in Python 2.4:: for x in reversed(sequence): ... # do something with x ... This won't touch your original sequence, but build a new copy with reversed order to iterate over. With Python 2.3, you can use an extended slice syntax:: for x in sequence[::-1]: ... # do something with x ... How do you remove duplicates from a list? ----------------------------------------- See the Python Cookbook for a long discussion of many ways to do this: https://code.activestate.com/recipes/52560/ If you don't mind reordering the list, sort it and then scan from the end of the list, deleting duplicates as you go:: if mylist: mylist.sort() last = mylist[-1] for i in range(len(mylist)-2, -1, -1): if last == mylist[i]: del mylist[i] else: last = mylist[i] If all elements of the list may be used as set keys (i.e. they are all :term:`hashable`) this is often faster :: mylist = list(set(mylist)) This converts the list into a set, thereby removing duplicates, and then back into a list. How do you make an array in Python? ----------------------------------- Use a list:: ["this", 1, "is", "an", "array"] Lists are equivalent to C or Pascal arrays in their time complexity; the primary difference is that a Python list can contain objects of many different types. The ``array`` module also provides methods for creating arrays of fixed types with compact representations, but they are slower to index than lists. Also note that the Numeric extensions and others define array-like structures with various characteristics as well. To get Lisp-style linked lists, you can emulate cons cells using tuples:: lisp_list = ("like", ("this", ("example", None) ) ) If mutability is desired, you could use lists instead of tuples. Here the analogue of lisp car is ``lisp_list[0]`` and the analogue of cdr is ``lisp_list[1]``. Only do this if you're sure you really need to, because it's usually a lot slower than using Python lists. .. _faq-multidimensional-list: How do I create a multidimensional list? ---------------------------------------- You probably tried to make a multidimensional array like this:: >>> A = [[None] * 2] * 3 This looks correct if you print it: .. testsetup:: A = [[None] * 2] * 3 .. doctest:: >>> A [[None, None], [None, None], [None, None]] But when you assign a value, it shows up in multiple places: .. testsetup:: A = [[None] * 2] * 3 .. doctest:: >>> A[0][0] = 5 >>> A [[5, None], [5, None], [5, None]] The reason is that replicating a list with ``*`` doesn't create copies, it only creates references to the existing objects. The ``*3`` creates a list containing 3 references to the same list of length two. Changes to one row will show in all rows, which is almost certainly not what you want. The suggested approach is to create a list of the desired length first and then fill in each element with a newly created list:: A = [None] * 3 for i in range(3): A[i] = [None] * 2 This generates a list containing 3 different lists of length two. You can also use a list comprehension:: w, h = 2, 3 A = [[None] * w for i in range(h)] Or, you can use an extension that provides a matrix datatype; `NumPy <http://www.numpy.org/>`_ is the best known. How do I apply a method to a sequence of objects? ------------------------------------------------- Use a list comprehension:: result = [obj.method() for obj in mylist] .. _faq-augmented-assignment-tuple-error: Why does a_tuple[i] += ['item'] raise an exception when the addition works? --------------------------------------------------------------------------- This is because of a combination of the fact that augmented assignment operators are *assignment* operators, and the difference between mutable and immutable objects in Python. This discussion applies in general when augmented assignment operators are applied to elements of a tuple that point to mutable objects, but we'll use a ``list`` and ``+=`` as our exemplar. If you wrote:: >>> a_tuple = (1, 2) >>> a_tuple[0] += 1 Traceback (most recent call last): ... TypeError: 'tuple' object does not support item assignment The reason for the exception should be immediately clear: ``1`` is added to the object ``a_tuple[0]`` points to (``1``), producing the result object, ``2``, but when we attempt to assign the result of the computation, ``2``, to element ``0`` of the tuple, we get an error because we can't change what an element of a tuple points to. Under the covers, what this augmented assignment statement is doing is approximately this:: >>> result = a_tuple[0] + 1 >>> a_tuple[0] = result Traceback (most recent call last): ... TypeError: 'tuple' object does not support item assignment It is the assignment part of the operation that produces the error, since a tuple is immutable. When you write something like:: >>> a_tuple = (['foo'], 'bar') >>> a_tuple[0] += ['item'] Traceback (most recent call last): ... TypeError: 'tuple' object does not support item assignment The exception is a bit more surprising, and even more surprising is the fact that even though there was an error, the append worked:: >>> a_tuple[0] ['foo', 'item'] To see why this happens, you need to know that (a) if an object implements an ``__iadd__`` magic method, it gets called when the ``+=`` augmented assignment is executed, and its return value is what gets used in the assignment statement; and (b) for lists, ``__iadd__`` is equivalent to calling ``extend`` on the list and returning the list. That's why we say that for lists, ``+=`` is a "shorthand" for ``list.extend``:: >>> a_list = [] >>> a_list += [1] >>> a_list [1] This is equivalent to:: >>> result = a_list.__iadd__([1]) >>> a_list = result The object pointed to by a_list has been mutated, and the pointer to the mutated object is assigned back to ``a_list``. The end result of the assignment is a no-op, since it is a pointer to the same object that ``a_list`` was previously pointing to, but the assignment still happens. Thus, in our tuple example what is happening is equivalent to:: >>> result = a_tuple[0].__iadd__(['item']) >>> a_tuple[0] = result Traceback (most recent call last): ... TypeError: 'tuple' object does not support item assignment The ``__iadd__`` succeeds, and thus the list is extended, but even though ``result`` points to the same object that ``a_tuple[0]`` already points to, that final assignment still results in an error, because tuples are immutable. Dictionaries ============ How can I get a dictionary to store and display its keys in a consistent order? ------------------------------------------------------------------------------- Use :class:`collections.OrderedDict`. I want to do a complicated sort: can you do a Schwartzian Transform in Python? ------------------------------------------------------------------------------ The technique, attributed to Randal Schwartz of the Perl community, sorts the elements of a list by a metric which maps each element to its "sort value". In Python, use the ``key`` argument for the :meth:`list.sort` method:: Isorted = L[:] Isorted.sort(key=lambda s: int(s[10:15])) How can I sort one list by values from another list? ---------------------------------------------------- Merge them into an iterator of tuples, sort the resulting list, and then pick out the element you want. :: >>> list1 = ["what", "I'm", "sorting", "by"] >>> list2 = ["something", "else", "to", "sort"] >>> pairs = zip(list1, list2) >>> pairs = sorted(pairs) >>> pairs [("I'm", 'else'), ('by', 'sort'), ('sorting', 'to'), ('what', 'something')] >>> result = [x[1] for x in pairs] >>> result ['else', 'sort', 'to', 'something'] An alternative for the last step is:: >>> result = [] >>> for p in pairs: result.append(p[1]) If you find this more legible, you might prefer to use this instead of the final list comprehension. However, it is almost twice as slow for long lists. Why? First, the ``append()`` operation has to reallocate memory, and while it uses some tricks to avoid doing that each time, it still has to do it occasionally, and that costs quite a bit. Second, the expression "result.append" requires an extra attribute lookup, and third, there's a speed reduction from having to make all those function calls. Objects ======= What is a class? ---------------- A class is the particular object type created by executing a class statement. Class objects are used as templates to create instance objects, which embody both the data (attributes) and code (methods) specific to a datatype. A class can be based on one or more other classes, called its base class(es). It then inherits the attributes and methods of its base classes. This allows an object model to be successively refined by inheritance. You might have a generic ``Mailbox`` class that provides basic accessor methods for a mailbox, and subclasses such as ``MboxMailbox``, ``MaildirMailbox``, ``OutlookMailbox`` that handle various specific mailbox formats. What is a method? ----------------- A method is a function on some object ``x`` that you normally call as ``x.name(arguments...)``. Methods are defined as functions inside the class definition:: class C: def meth(self, arg): return arg * 2 + self.attribute What is self? ------------- Self is merely a conventional name for the first argument of a method. A method defined as ``meth(self, a, b, c)`` should be called as ``x.meth(a, b, c)`` for some instance ``x`` of the class in which the definition occurs; the called method will think it is called as ``meth(x, a, b, c)``. See also :ref:`why-self`. How do I check if an object is an instance of a given class or of a subclass of it? ----------------------------------------------------------------------------------- Use the built-in function ``isinstance(obj, cls)``. You can check if an object is an instance of any of a number of classes by providing a tuple instead of a single class, e.g. ``isinstance(obj, (class1, class2, ...))``, and can also check whether an object is one of Python's built-in types, e.g. ``isinstance(obj, str)`` or ``isinstance(obj, (int, float, complex))``. Note that most programs do not use :func:`isinstance` on user-defined classes very often. If you are developing the classes yourself, a more proper object-oriented style is to define methods on the classes that encapsulate a particular behaviour, instead of checking the object's class and doing a different thing based on what class it is. For example, if you have a function that does something:: def search(obj): if isinstance(obj, Mailbox): ... # code to search a mailbox elif isinstance(obj, Document): ... # code to search a document elif ... A better approach is to define a ``search()`` method on all the classes and just call it:: class Mailbox: def search(self): ... # code to search a mailbox class Document: def search(self): ... # code to search a document obj.search() What is delegation? ------------------- Delegation is an object oriented technique (also called a design pattern). Let's say you have an object ``x`` and want to change the behaviour of just one of its methods. You can create a new class that provides a new implementation of the method you're interested in changing and delegates all other methods to the corresponding method of ``x``. Python programmers can easily implement delegation. For example, the following class implements a class that behaves like a file but converts all written data to uppercase:: class UpperOut: def __init__(self, outfile): self._outfile = outfile def write(self, s): self._outfile.write(s.upper()) def __getattr__(self, name): return getattr(self._outfile, name) Here the ``UpperOut`` class redefines the ``write()`` method to convert the argument string to uppercase before calling the underlying ``self.__outfile.write()`` method. All other methods are delegated to the underlying ``self.__outfile`` object. The delegation is accomplished via the ``__getattr__`` method; consult :ref:`the language reference <attribute-access>` for more information about controlling attribute access. Note that for more general cases delegation can get trickier. When attributes must be set as well as retrieved, the class must define a :meth:`__setattr__` method too, and it must do so carefully. The basic implementation of :meth:`__setattr__` is roughly equivalent to the following:: class X: ... def __setattr__(self, name, value): self.__dict__[name] = value ... Most :meth:`__setattr__` implementations must modify ``self.__dict__`` to store local state for self without causing an infinite recursion. How do I call a method defined in a base class from a derived class that overrides it? -------------------------------------------------------------------------------------- Use the built-in :func:`super` function:: class Derived(Base): def meth(self): super(Derived, self).meth() For version prior to 3.0, you may be using classic classes: For a class definition such as ``class Derived(Base): ...`` you can call method ``meth()`` defined in ``Base`` (or one of ``Base``'s base classes) as ``Base.meth(self, arguments...)``. Here, ``Base.meth`` is an unbound method, so you need to provide the ``self`` argument. How can I organize my code to make it easier to change the base class? ---------------------------------------------------------------------- You could define an alias for the base class, assign the real base class to it before your class definition, and use the alias throughout your class. Then all you have to change is the value assigned to the alias. Incidentally, this trick is also handy if you want to decide dynamically (e.g. depending on availability of resources) which base class to use. Example:: BaseAlias = <real base class> class Derived(BaseAlias): def meth(self): BaseAlias.meth(self) ... How do I create static class data and static class methods? ----------------------------------------------------------- Both static data and static methods (in the sense of C++ or Java) are supported in Python. For static data, simply define a class attribute. To assign a new value to the attribute, you have to explicitly use the class name in the assignment:: class C: count = 0 # number of times C.__init__ called def __init__(self): C.count = C.count + 1 def getcount(self): return C.count # or return self.count ``c.count`` also refers to ``C.count`` for any ``c`` such that ``isinstance(c, C)`` holds, unless overridden by ``c`` itself or by some class on the base-class search path from ``c.__class__`` back to ``C``. Caution: within a method of C, an assignment like ``self.count = 42`` creates a new and unrelated instance named "count" in ``self``'s own dict. Rebinding of a class-static data name must always specify the class whether inside a method or not:: C.count = 314 Static methods are possible:: class C: @staticmethod def static(arg1, arg2, arg3): # No 'self' parameter! ... However, a far more straightforward way to get the effect of a static method is via a simple module-level function:: def getcount(): return C.count If your code is structured so as to define one class (or tightly related class hierarchy) per module, this supplies the desired encapsulation. How can I overload constructors (or methods) in Python? ------------------------------------------------------- This answer actually applies to all methods, but the question usually comes up first in the context of constructors. In C++ you'd write .. code-block:: c class C { C() { cout << "No arguments\n"; } C(int i) { cout << "Argument is " << i << "\n"; } } In Python you have to write a single constructor that catches all cases using default arguments. For example:: class C: def __init__(self, i=None): if i is None: print("No arguments") else: print("Argument is", i) This is not entirely equivalent, but close enough in practice. You could also try a variable-length argument list, e.g. :: def __init__(self, *args): ... The same approach works for all method definitions. I try to use __spam and I get an error about _SomeClassName__spam. ------------------------------------------------------------------ Variable names with double leading underscores are "mangled" to provide a simple but effective way to define class private variables. Any identifier of the form ``__spam`` (at least two leading underscores, at most one trailing underscore) is textually replaced with ``_classname__spam``, where ``classname`` is the current class name with any leading underscores stripped. This doesn't guarantee privacy: an outside user can still deliberately access the "_classname__spam" attribute, and private values are visible in the object's ``__dict__``. Many Python programmers never bother to use private variable names at all. My class defines __del__ but it is not called when I delete the object. ----------------------------------------------------------------------- There are several possible reasons for this. The del statement does not necessarily call :meth:`__del__` -- it simply decrements the object's reference count, and if this reaches zero :meth:`__del__` is called. If your data structures contain circular links (e.g. a tree where each child has a parent reference and each parent has a list of children) the reference counts will never go back to zero. Once in a while Python runs an algorithm to detect such cycles, but the garbage collector might run some time after the last reference to your data structure vanishes, so your :meth:`__del__` method may be called at an inconvenient and random time. This is inconvenient if you're trying to reproduce a problem. Worse, the order in which object's :meth:`__del__` methods are executed is arbitrary. You can run :func:`gc.collect` to force a collection, but there *are* pathological cases where objects will never be collected. Despite the cycle collector, it's still a good idea to define an explicit ``close()`` method on objects to be called whenever you're done with them. The ``close()`` method can then remove attributes that refer to subobjects. Don't call :meth:`__del__` directly -- :meth:`__del__` should call ``close()`` and ``close()`` should make sure that it can be called more than once for the same object. Another way to avoid cyclical references is to use the :mod:`weakref` module, which allows you to point to objects without incrementing their reference count. Tree data structures, for instance, should use weak references for their parent and sibling references (if they need them!). .. XXX relevant for Python 3? If the object has ever been a local variable in a function that caught an expression in an except clause, chances are that a reference to the object still exists in that function's stack frame as contained in the stack trace. Normally, calling :func:`sys.exc_clear` will take care of this by clearing the last recorded exception. Finally, if your :meth:`__del__` method raises an exception, a warning message is printed to :data:`sys.stderr`. How do I get a list of all instances of a given class? ------------------------------------------------------ Python does not keep track of all instances of a class (or of a built-in type). You can program the class's constructor to keep track of all instances by keeping a list of weak references to each instance. Why does the result of ``id()`` appear to be not unique? -------------------------------------------------------- The :func:`id` builtin returns an integer that is guaranteed to be unique during the lifetime of the object. Since in CPython, this is the object's memory address, it happens frequently that after an object is deleted from memory, the next freshly created object is allocated at the same position in memory. This is illustrated by this example: >>> id(1000) # doctest: +SKIP 13901272 >>> id(2000) # doctest: +SKIP 13901272 The two ids belong to different integer objects that are created before, and deleted immediately after execution of the ``id()`` call. To be sure that objects whose id you want to examine are still alive, create another reference to the object: >>> a = 1000; b = 2000 >>> id(a) # doctest: +SKIP 13901272 >>> id(b) # doctest: +SKIP 13891296 Modules ======= How do I create a .pyc file? ---------------------------- When a module is imported for the first time (or when the source file has changed since the current compiled file was created) a ``.pyc`` file containing the compiled code should be created in a ``__pycache__`` subdirectory of the directory containing the ``.py`` file. The ``.pyc`` file will have a filename that starts with the same name as the ``.py`` file, and ends with ``.pyc``, with a middle component that depends on the particular ``python`` binary that created it. (See :pep:`3147` for details.) One reason that a ``.pyc`` file may not be created is a permissions problem with the directory containing the source file, meaning that the ``__pycache__`` subdirectory cannot be created. This can happen, for example, if you develop as one user but run as another, such as if you are testing with a web server. Unless the :envvar:`PYTHONDONTWRITEBYTECODE` environment variable is set, creation of a .pyc file is automatic if you're importing a module and Python has the ability (permissions, free space, etc...) to create a ``__pycache__`` subdirectory and write the compiled module to that subdirectory. Running Python on a top level script is not considered an import and no ``.pyc`` will be created. For example, if you have a top-level module ``foo.py`` that imports another module ``xyz.py``, when you run ``foo`` (by typing ``python foo.py`` as a shell command), a ``.pyc`` will be created for ``xyz`` because ``xyz`` is imported, but no ``.pyc`` file will be created for ``foo`` since ``foo.py`` isn't being imported. If you need to create a ``.pyc`` file for ``foo`` -- that is, to create a ``.pyc`` file for a module that is not imported -- you can, using the :mod:`py_compile` and :mod:`compileall` modules. The :mod:`py_compile` module can manually compile any module. One way is to use the ``compile()`` function in that module interactively:: >>> import py_compile >>> py_compile.compile('foo.py') # doctest: +SKIP This will write the ``.pyc`` to a ``__pycache__`` subdirectory in the same location as ``foo.py`` (or you can override that with the optional parameter ``cfile``). You can also automatically compile all files in a directory or directories using the :mod:`compileall` module. You can do it from the shell prompt by running ``compileall.py`` and providing the path of a directory containing Python files to compile:: python -m compileall . How do I find the current module name? -------------------------------------- A module can find out its own module name by looking at the predefined global variable ``__name__``. If this has the value ``'__main__'``, the program is running as a script. Many modules that are usually used by importing them also provide a command-line interface or a self-test, and only execute this code after checking ``__name__``:: def main(): print('Running test...') ... if __name__ == '__main__': main() How can I have modules that mutually import each other? ------------------------------------------------------- Suppose you have the following modules: foo.py:: from bar import bar_var foo_var = 1 bar.py:: from foo import foo_var bar_var = 2 The problem is that the interpreter will perform the following steps: * main imports foo * Empty globals for foo are created * foo is compiled and starts executing * foo imports bar * Empty globals for bar are created * bar is compiled and starts executing * bar imports foo (which is a no-op since there already is a module named foo) * bar.foo_var = foo.foo_var The last step fails, because Python isn't done with interpreting ``foo`` yet and the global symbol dictionary for ``foo`` is still empty. The same thing happens when you use ``import foo``, and then try to access ``foo.foo_var`` in global code. There are (at least) three possible workarounds for this problem. Guido van Rossum recommends avoiding all uses of ``from <module> import ...``, and placing all code inside functions. Initializations of global variables and class variables should use constants or built-in functions only. This means everything from an imported module is referenced as ``<module>.<name>``. Jim Roskind suggests performing steps in the following order in each module: * exports (globals, functions, and classes that don't need imported base classes) * ``import`` statements * active code (including globals that are initialized from imported values). van Rossum doesn't like this approach much because the imports appear in a strange place, but it does work. Matthias Urlichs recommends restructuring your code so that the recursive import is not necessary in the first place. These solutions are not mutually exclusive. __import__('x.y.z') returns <module 'x'>; how do I get z? --------------------------------------------------------- Consider using the convenience function :func:`~importlib.import_module` from :mod:`importlib` instead:: z = importlib.import_module('x.y.z') When I edit an imported module and reimport it, the changes don't show up. Why does this happen? ------------------------------------------------------------------------------------------------- For reasons of efficiency as well as consistency, Python only reads the module file on the first time a module is imported. If it didn't, in a program consisting of many modules where each one imports the same basic module, the basic module would be parsed and re-parsed many times. To force re-reading of a changed module, do this:: import importlib import modname importlib.reload(modname) Warning: this technique is not 100% fool-proof. In particular, modules containing statements like :: from modname import some_objects will continue to work with the old version of the imported objects. If the module contains class definitions, existing class instances will *not* be updated to use the new class definition. This can result in the following paradoxical behaviour: >>> import importlib >>> import cls >>> c = cls.C() # Create an instance of C >>> importlib.reload(cls) <module 'cls' from 'cls.py'> >>> isinstance(c, cls.C) # isinstance is false?!? False The nature of the problem is made clear if you print out the "identity" of the class objects: >>> hex(id(c.__class__)) '0x7352a0' >>> hex(id(cls.C)) '0x4198d0' PK����� 3]Vv������faq/index.rst.txtnu�[��������.. _faq-index: ################################### Python Frequently Asked Questions ################################### .. toctree:: :maxdepth: 1 general.rst programming.rst design.rst library.rst extending.rst windows.rst gui.rst installed.rst PK����� 3]2x������install/index.rst.txtnu�[��������.. highlightlang:: none .. _install-index: ******************************************** Installing Python Modules (Legacy version) ******************************************** :Author: Greg Ward .. TODO: Fill in XXX comments .. seealso:: :ref:`installing-index` The up to date module installation documentations .. The audience for this document includes people who don't know anything about Python and aren't about to learn the language just in order to install and maintain it for their users, i.e. system administrators. Thus, I have to be sure to explain the basics at some point: sys.path and PYTHONPATH at least. Should probably give pointers to other docs on "import site", PYTHONSTARTUP, PYTHONHOME, etc. Finally, it might be useful to include all the material from my "Care and Feeding of a Python Installation" talk in here somewhere. Yow! This document describes the Python Distribution Utilities ("Distutils") from the end-user's point-of-view, describing how to extend the capabilities of a standard Python installation by building and installing third-party Python modules and extensions. .. note:: This guide only covers the basic tools for building and distributing extensions that are provided as part of this version of Python. Third party tools offer easier to use and more secure alternatives. Refer to the `quick recommendations section <https://packaging.python.org/en/latest/current/>`__ in the Python Packaging User Guide for more information. .. _inst-intro: Introduction ============ Although Python's extensive standard library covers many programming needs, there often comes a time when you need to add some new functionality to your Python installation in the form of third-party modules. This might be necessary to support your own programming, or to support an application that you want to use and that happens to be written in Python. In the past, there has been little support for adding third-party modules to an existing Python installation. With the introduction of the Python Distribution Utilities (Distutils for short) in Python 2.0, this changed. This document is aimed primarily at the people who need to install third-party Python modules: end-users and system administrators who just need to get some Python application running, and existing Python programmers who want to add some new goodies to their toolbox. You don't need to know Python to read this document; there will be some brief forays into using Python's interactive mode to explore your installation, but that's it. If you're looking for information on how to distribute your own Python modules so that others may use them, see the :ref:`distutils-index` manual. :ref:`debug-setup-script` may also be of interest. .. _inst-trivial-install: Best case: trivial installation ------------------------------- In the best case, someone will have prepared a special version of the module distribution you want to install that is targeted specifically at your platform and is installed just like any other software on your platform. For example, the module developer might make an executable installer available for Windows users, an RPM package for users of RPM-based Linux systems (Red Hat, SuSE, Mandrake, and many others), a Debian package for users of Debian-based Linux systems, and so forth. In that case, you would download the installer appropriate to your platform and do the obvious thing with it: run it if it's an executable installer, ``rpm --install`` it if it's an RPM, etc. You don't need to run Python or a setup script, you don't need to compile anything---you might not even need to read any instructions (although it's always a good idea to do so anyway). Of course, things will not always be that easy. You might be interested in a module distribution that doesn't have an easy-to-use installer for your platform. In that case, you'll have to start with the source distribution released by the module's author/maintainer. Installing from a source distribution is not too hard, as long as the modules are packaged in the standard way. The bulk of this document is about building and installing modules from standard source distributions. .. _inst-new-standard: The new standard: Distutils --------------------------- If you download a module source distribution, you can tell pretty quickly if it was packaged and distributed in the standard way, i.e. using the Distutils. First, the distribution's name and version number will be featured prominently in the name of the downloaded archive, e.g. :file:`foo-1.0.tar.gz` or :file:`widget-0.9.7.zip`. Next, the archive will unpack into a similarly-named directory: :file:`foo-1.0` or :file:`widget-0.9.7`. Additionally, the distribution will contain a setup script :file:`setup.py`, and a file named :file:`README.txt` or possibly just :file:`README`, which should explain that building and installing the module distribution is a simple matter of running one command from a terminal:: python setup.py install For Windows, this command should be run from a command prompt window (:menuselection:`Start --> Accessories`):: setup.py install If all these things are true, then you already know how to build and install the modules you've just downloaded: Run the command above. Unless you need to install things in a non-standard way or customize the build process, you don't really need this manual. Or rather, the above command is everything you need to get out of this manual. .. _inst-standard-install: Standard Build and Install ========================== As described in section :ref:`inst-new-standard`, building and installing a module distribution using the Distutils is usually one simple command to run from a terminal:: python setup.py install .. _inst-platform-variations: Platform variations ------------------- You should always run the setup command from the distribution root directory, i.e. the top-level subdirectory that the module source distribution unpacks into. For example, if you've just downloaded a module source distribution :file:`foo-1.0.tar.gz` onto a Unix system, the normal thing to do is:: gunzip -c foo-1.0.tar.gz | tar xf - # unpacks into directory foo-1.0 cd foo-1.0 python setup.py install On Windows, you'd probably download :file:`foo-1.0.zip`. If you downloaded the archive file to :file:`C:\\Temp`, then it would unpack into :file:`C:\\Temp\\foo-1.0`; you can use either an archive manipulator with a graphical user interface (such as WinZip) or a command-line tool (such as :program:`unzip` or :program:`pkunzip`) to unpack the archive. Then, open a command prompt window and run:: cd c:\Temp\foo-1.0 python setup.py install .. _inst-splitting-up: Splitting the job up -------------------- Running ``setup.py install`` builds and installs all modules in one run. If you prefer to work incrementally---especially useful if you want to customize the build process, or if things are going wrong---you can use the setup script to do one thing at a time. This is particularly helpful when the build and install will be done by different users---for example, you might want to build a module distribution and hand it off to a system administrator for installation (or do it yourself, with super-user privileges). For example, you can build everything in one step, and then install everything in a second step, by invoking the setup script twice:: python setup.py build python setup.py install If you do this, you will notice that running the :command:`install` command first runs the :command:`build` command, which---in this case---quickly notices that it has nothing to do, since everything in the :file:`build` directory is up-to-date. You may not need this ability to break things down often if all you do is install modules downloaded off the 'net, but it's very handy for more advanced tasks. If you get into distributing your own Python modules and extensions, you'll run lots of individual Distutils commands on their own. .. _inst-how-build-works: How building works ------------------ As implied above, the :command:`build` command is responsible for putting the files to install into a *build directory*. By default, this is :file:`build` under the distribution root; if you're excessively concerned with speed, or want to keep the source tree pristine, you can change the build directory with the :option:`!--build-base` option. For example:: python setup.py build --build-base=/path/to/pybuild/foo-1.0 (Or you could do this permanently with a directive in your system or personal Distutils configuration file; see section :ref:`inst-config-files`.) Normally, this isn't necessary. The default layout for the build tree is as follows:: --- build/ --- lib/ or --- build/ --- lib.<plat>/ temp.<plat>/ where ``<plat>`` expands to a brief description of the current OS/hardware platform and Python version. The first form, with just a :file:`lib` directory, is used for "pure module distributions"---that is, module distributions that include only pure Python modules. If a module distribution contains any extensions (modules written in C/C++), then the second form, with two ``<plat>`` directories, is used. In that case, the :file:`temp.{plat}` directory holds temporary files generated by the compile/link process that don't actually get installed. In either case, the :file:`lib` (or :file:`lib.{plat}`) directory contains all Python modules (pure Python and extensions) that will be installed. In the future, more directories will be added to handle Python scripts, documentation, binary executables, and whatever else is needed to handle the job of installing Python modules and applications. .. _inst-how-install-works: How installation works ---------------------- After the :command:`build` command runs (whether you run it explicitly, or the :command:`install` command does it for you), the work of the :command:`install` command is relatively simple: all it has to do is copy everything under :file:`build/lib` (or :file:`build/lib.{plat}`) to your chosen installation directory. If you don't choose an installation directory---i.e., if you just run ``setup.py install``\ ---then the :command:`install` command installs to the standard location for third-party Python modules. This location varies by platform and by how you built/installed Python itself. On Unix (and Mac OS X, which is also Unix-based), it also depends on whether the module distribution being installed is pure Python or contains extensions ("non-pure"): .. tabularcolumns:: |l|l|l|l| +-----------------+-----------------------------------------------------+--------------------------------------------------+-------+ | Platform | Standard installation location | Default value | Notes | +=================+=====================================================+==================================================+=======+ | Unix (pure) | :file:`{prefix}/lib/python{X.Y}/site-packages` | :file:`/usr/local/lib/python{X.Y}/site-packages` | \(1) | +-----------------+-----------------------------------------------------+--------------------------------------------------+-------+ | Unix (non-pure) | :file:`{exec-prefix}/lib/python{X.Y}/site-packages` | :file:`/usr/local/lib/python{X.Y}/site-packages` | \(1) | +-----------------+-----------------------------------------------------+--------------------------------------------------+-------+ | Windows | :file:`{prefix}\\Lib\\site-packages` | :file:`C:\\Python{XY}\\Lib\\site-packages` | \(2) | +-----------------+-----------------------------------------------------+--------------------------------------------------+-------+ Notes: (1) Most Linux distributions include Python as a standard part of the system, so :file:`{prefix}` and :file:`{exec-prefix}` are usually both :file:`/usr` on Linux. If you build Python yourself on Linux (or any Unix-like system), the default :file:`{prefix}` and :file:`{exec-prefix}` are :file:`/usr/local`. (2) The default installation directory on Windows was :file:`C:\\Program Files\\Python` under Python 1.6a1, 1.5.2, and earlier. :file:`{prefix}` and :file:`{exec-prefix}` stand for the directories that Python is installed to, and where it finds its libraries at run-time. They are always the same under Windows, and very often the same under Unix and Mac OS X. You can find out what your Python installation uses for :file:`{prefix}` and :file:`{exec-prefix}` by running Python in interactive mode and typing a few simple commands. Under Unix, just type ``python`` at the shell prompt. Under Windows, choose :menuselection:`Start --> Programs --> Python X.Y --> Python (command line)`. Once the interpreter is started, you type Python code at the prompt. For example, on my Linux system, I type the three Python statements shown below, and get the output as shown, to find out my :file:`{prefix}` and :file:`{exec-prefix}`: .. code-block:: pycon Python 2.4 (#26, Aug 7 2004, 17:19:02) Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.prefix '/usr' >>> sys.exec_prefix '/usr' A few other placeholders are used in this document: :file:`{X.Y}` stands for the version of Python, for example ``3.2``; :file:`{abiflags}` will be replaced by the value of :data:`sys.abiflags` or the empty string for platforms which don't define ABI flags; :file:`{distname}` will be replaced by the name of the module distribution being installed. Dots and capitalization are important in the paths; for example, a value that uses ``python3.2`` on UNIX will typically use ``Python32`` on Windows. If you don't want to install modules to the standard location, or if you don't have permission to write there, then you need to read about alternate installations in section :ref:`inst-alt-install`. If you want to customize your installation directories more heavily, see section :ref:`inst-custom-install` on custom installations. .. _inst-alt-install: Alternate Installation ====================== Often, it is necessary or desirable to install modules to a location other than the standard location for third-party Python modules. For example, on a Unix system you might not have permission to write to the standard third-party module directory. Or you might wish to try out a module before making it a standard part of your local Python installation. This is especially true when upgrading a distribution already present: you want to make sure your existing base of scripts still works with the new version before actually upgrading. The Distutils :command:`install` command is designed to make installing module distributions to an alternate location simple and painless. The basic idea is that you supply a base directory for the installation, and the :command:`install` command picks a set of directories (called an *installation scheme*) under this base directory in which to install files. The details differ across platforms, so read whichever of the following sections applies to you. Note that the various alternate installation schemes are mutually exclusive: you can pass ``--user``, or ``--home``, or ``--prefix`` and ``--exec-prefix``, or ``--install-base`` and ``--install-platbase``, but you can't mix from these groups. .. _inst-alt-install-user: Alternate installation: the user scheme --------------------------------------- This scheme is designed to be the most convenient solution for users that don't have write permission to the global site-packages directory or don't want to install into it. It is enabled with a simple option:: python setup.py install --user Files will be installed into subdirectories of :data:`site.USER_BASE` (written as :file:`{userbase}` hereafter). This scheme installs pure Python modules and extension modules in the same location (also known as :data:`site.USER_SITE`). Here are the values for UNIX, including Mac OS X: =============== =========================================================== Type of file Installation directory =============== =========================================================== modules :file:`{userbase}/lib/python{X.Y}/site-packages` scripts :file:`{userbase}/bin` data :file:`{userbase}` C headers :file:`{userbase}/include/python{X.Y}{abiflags}/{distname}` =============== =========================================================== And here are the values used on Windows: =============== =========================================================== Type of file Installation directory =============== =========================================================== modules :file:`{userbase}\\Python{XY}\\site-packages` scripts :file:`{userbase}\\Python{XY}\\Scripts` data :file:`{userbase}` C headers :file:`{userbase}\\Python{XY}\\Include\\{distname}` =============== =========================================================== The advantage of using this scheme compared to the other ones described below is that the user site-packages directory is under normal conditions always included in :data:`sys.path` (see :mod:`site` for more information), which means that there is no additional step to perform after running the :file:`setup.py` script to finalize the installation. The :command:`build_ext` command also has a ``--user`` option to add :file:`{userbase}/include` to the compiler search path for header files and :file:`{userbase}/lib` to the compiler search path for libraries as well as to the runtime search path for shared C libraries (rpath). .. _inst-alt-install-home: Alternate installation: the home scheme --------------------------------------- The idea behind the "home scheme" is that you build and maintain a personal stash of Python modules. This scheme's name is derived from the idea of a "home" directory on Unix, since it's not unusual for a Unix user to make their home directory have a layout similar to :file:`/usr/` or :file:`/usr/local/`. This scheme can be used by anyone, regardless of the operating system they are installing for. Installing a new module distribution is as simple as :: python setup.py install --home=<dir> where you can supply any directory you like for the :option:`!--home` option. On Unix, lazy typists can just type a tilde (``~``); the :command:`install` command will expand this to your home directory:: python setup.py install --home=~ To make Python find the distributions installed with this scheme, you may have to :ref:`modify Python's search path <inst-search-path>` or edit :mod:`sitecustomize` (see :mod:`site`) to call :func:`site.addsitedir` or edit :data:`sys.path`. The :option:`!--home` option defines the installation base directory. Files are installed to the following directories under the installation base as follows: =============== =========================================================== Type of file Installation directory =============== =========================================================== modules :file:`{home}/lib/python` scripts :file:`{home}/bin` data :file:`{home}` C headers :file:`{home}/include/python/{distname}` =============== =========================================================== (Mentally replace slashes with backslashes if you're on Windows.) .. _inst-alt-install-prefix-unix: Alternate installation: Unix (the prefix scheme) ------------------------------------------------ The "prefix scheme" is useful when you wish to use one Python installation to perform the build/install (i.e., to run the setup script), but install modules into the third-party module directory of a different Python installation (or something that looks like a different Python installation). If this sounds a trifle unusual, it is---that's why the user and home schemes come before. However, there are at least two known cases where the prefix scheme will be useful. First, consider that many Linux distributions put Python in :file:`/usr`, rather than the more traditional :file:`/usr/local`. This is entirely appropriate, since in those cases Python is part of "the system" rather than a local add-on. However, if you are installing Python modules from source, you probably want them to go in :file:`/usr/local/lib/python2.{X}` rather than :file:`/usr/lib/python2.{X}`. This can be done with :: /usr/bin/python setup.py install --prefix=/usr/local Another possibility is a network filesystem where the name used to write to a remote directory is different from the name used to read it: for example, the Python interpreter accessed as :file:`/usr/local/bin/python` might search for modules in :file:`/usr/local/lib/python2.{X}`, but those modules would have to be installed to, say, :file:`/mnt/{@server}/export/lib/python2.{X}`. This could be done with :: /usr/local/bin/python setup.py install --prefix=/mnt/@server/export In either case, the :option:`!--prefix` option defines the installation base, and the :option:`!--exec-prefix` option defines the platform-specific installation base, which is used for platform-specific files. (Currently, this just means non-pure module distributions, but could be expanded to C libraries, binary executables, etc.) If :option:`!--exec-prefix` is not supplied, it defaults to :option:`!--prefix`. Files are installed as follows: ================= ========================================================== Type of file Installation directory ================= ========================================================== Python modules :file:`{prefix}/lib/python{X.Y}/site-packages` extension modules :file:`{exec-prefix}/lib/python{X.Y}/site-packages` scripts :file:`{prefix}/bin` data :file:`{prefix}` C headers :file:`{prefix}/include/python{X.Y}{abiflags}/{distname}` ================= ========================================================== There is no requirement that :option:`!--prefix` or :option:`!--exec-prefix` actually point to an alternate Python installation; if the directories listed above do not already exist, they are created at installation time. Incidentally, the real reason the prefix scheme is important is simply that a standard Unix installation uses the prefix scheme, but with :option:`!--prefix` and :option:`!--exec-prefix` supplied by Python itself as ``sys.prefix`` and ``sys.exec_prefix``. Thus, you might think you'll never use the prefix scheme, but every time you run ``python setup.py install`` without any other options, you're using it. Note that installing extensions to an alternate Python installation has no effect on how those extensions are built: in particular, the Python header files (:file:`Python.h` and friends) installed with the Python interpreter used to run the setup script will be used in compiling extensions. It is your responsibility to ensure that the interpreter used to run extensions installed in this way is compatible with the interpreter used to build them. The best way to do this is to ensure that the two interpreters are the same version of Python (possibly different builds, or possibly copies of the same build). (Of course, if your :option:`!--prefix` and :option:`!--exec-prefix` don't even point to an alternate Python installation, this is immaterial.) .. _inst-alt-install-prefix-windows: Alternate installation: Windows (the prefix scheme) --------------------------------------------------- Windows has no concept of a user's home directory, and since the standard Python installation under Windows is simpler than under Unix, the :option:`!--prefix` option has traditionally been used to install additional packages in separate locations on Windows. :: python setup.py install --prefix="\Temp\Python" to install modules to the :file:`\\Temp\\Python` directory on the current drive. The installation base is defined by the :option:`!--prefix` option; the :option:`!--exec-prefix` option is not supported under Windows, which means that pure Python modules and extension modules are installed into the same location. Files are installed as follows: =============== ========================================================== Type of file Installation directory =============== ========================================================== modules :file:`{prefix}\\Lib\\site-packages` scripts :file:`{prefix}\\Scripts` data :file:`{prefix}` C headers :file:`{prefix}\\Include\\{distname}` =============== ========================================================== .. _inst-custom-install: Custom Installation =================== Sometimes, the alternate installation schemes described in section :ref:`inst-alt-install` just don't do what you want. You might want to tweak just one or two directories while keeping everything under the same base directory, or you might want to completely redefine the installation scheme. In either case, you're creating a *custom installation scheme*. To create a custom installation scheme, you start with one of the alternate schemes and override some of the installation directories used for the various types of files, using these options: ====================== ======================= Type of file Override option ====================== ======================= Python modules ``--install-purelib`` extension modules ``--install-platlib`` all modules ``--install-lib`` scripts ``--install-scripts`` data ``--install-data`` C headers ``--install-headers`` ====================== ======================= These override options can be relative, absolute, or explicitly defined in terms of one of the installation base directories. (There are two installation base directories, and they are normally the same--- they only differ when you use the Unix "prefix scheme" and supply different ``--prefix`` and ``--exec-prefix`` options; using ``--install-lib`` will override values computed or given for ``--install-purelib`` and ``--install-platlib``, and is recommended for schemes that don't make a difference between Python and extension modules.) For example, say you're installing a module distribution to your home directory under Unix---but you want scripts to go in :file:`~/scripts` rather than :file:`~/bin`. As you might expect, you can override this directory with the :option:`!--install-scripts` option; in this case, it makes most sense to supply a relative path, which will be interpreted relative to the installation base directory (your home directory, in this case):: python setup.py install --home=~ --install-scripts=scripts Another Unix example: suppose your Python installation was built and installed with a prefix of :file:`/usr/local/python`, so under a standard installation scripts will wind up in :file:`/usr/local/python/bin`. If you want them in :file:`/usr/local/bin` instead, you would supply this absolute directory for the :option:`!--install-scripts` option:: python setup.py install --install-scripts=/usr/local/bin (This performs an installation using the "prefix scheme," where the prefix is whatever your Python interpreter was installed with--- :file:`/usr/local/python` in this case.) If you maintain Python on Windows, you might want third-party modules to live in a subdirectory of :file:`{prefix}`, rather than right in :file:`{prefix}` itself. This is almost as easy as customizing the script installation directory ---you just have to remember that there are two types of modules to worry about, Python and extension modules, which can conveniently be both controlled by one option:: python setup.py install --install-lib=Site The specified installation directory is relative to :file:`{prefix}`. Of course, you also have to ensure that this directory is in Python's module search path, such as by putting a :file:`.pth` file in a site directory (see :mod:`site`). See section :ref:`inst-search-path` to find out how to modify Python's search path. If you want to define an entire installation scheme, you just have to supply all of the installation directory options. The recommended way to do this is to supply relative paths; for example, if you want to maintain all Python module-related files under :file:`python` in your home directory, and you want a separate directory for each platform that you use your home directory from, you might define the following installation scheme:: python setup.py install --home=~ \ --install-purelib=python/lib \ --install-platlib=python/lib.$PLAT \ --install-scripts=python/scripts --install-data=python/data or, equivalently, :: python setup.py install --home=~/python \ --install-purelib=lib \ --install-platlib='lib.$PLAT' \ --install-scripts=scripts --install-data=data ``$PLAT`` is not (necessarily) an environment variable---it will be expanded by the Distutils as it parses your command line options, just as it does when parsing your configuration file(s). Obviously, specifying the entire installation scheme every time you install a new module distribution would be very tedious. Thus, you can put these options into your Distutils config file (see section :ref:`inst-config-files`): .. code-block:: ini [install] install-base=$HOME install-purelib=python/lib install-platlib=python/lib.$PLAT install-scripts=python/scripts install-data=python/data or, equivalently, .. code-block:: ini [install] install-base=$HOME/python install-purelib=lib install-platlib=lib.$PLAT install-scripts=scripts install-data=data Note that these two are *not* equivalent if you supply a different installation base directory when you run the setup script. For example, :: python setup.py install --install-base=/tmp would install pure modules to :file:`/tmp/python/lib` in the first case, and to :file:`/tmp/lib` in the second case. (For the second case, you probably want to supply an installation base of :file:`/tmp/python`.) You probably noticed the use of ``$HOME`` and ``$PLAT`` in the sample configuration file input. These are Distutils configuration variables, which bear a strong resemblance to environment variables. In fact, you can use environment variables in config files on platforms that have such a notion but the Distutils additionally define a few extra variables that may not be in your environment, such as ``$PLAT``. (And of course, on systems that don't have environment variables, such as Mac OS 9, the configuration variables supplied by the Distutils are the only ones you can use.) See section :ref:`inst-config-files` for details. .. note:: When a :ref:`virtual environment <venv-def>` is activated, any options that change the installation path will be ignored from all distutils configuration files to prevent inadvertently installing projects outside of the virtual environment. .. XXX need some Windows examples---when would custom installation schemes be needed on those platforms? .. XXX Move this to Doc/using .. _inst-search-path: Modifying Python's Search Path ------------------------------ When the Python interpreter executes an :keyword:`import` statement, it searches for both Python code and extension modules along a search path. A default value for the path is configured into the Python binary when the interpreter is built. You can determine the path by importing the :mod:`sys` module and printing the value of ``sys.path``. :: $ python Python 2.2 (#11, Oct 3 2002, 13:31:27) [GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-112)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path ['', '/usr/local/lib/python2.3', '/usr/local/lib/python2.3/plat-linux2', '/usr/local/lib/python2.3/lib-tk', '/usr/local/lib/python2.3/lib-dynload', '/usr/local/lib/python2.3/site-packages'] >>> The null string in ``sys.path`` represents the current working directory. The expected convention for locally installed packages is to put them in the :file:`{...}/site-packages/` directory, but you may want to install Python modules into some arbitrary directory. For example, your site may have a convention of keeping all software related to the web server under :file:`/www`. Add-on Python modules might then belong in :file:`/www/python`, and in order to import them, this directory must be added to ``sys.path``. There are several different ways to add the directory. The most convenient way is to add a path configuration file to a directory that's already on Python's path, usually to the :file:`.../site-packages/` directory. Path configuration files have an extension of :file:`.pth`, and each line must contain a single path that will be appended to ``sys.path``. (Because the new paths are appended to ``sys.path``, modules in the added directories will not override standard modules. This means you can't use this mechanism for installing fixed versions of standard modules.) Paths can be absolute or relative, in which case they're relative to the directory containing the :file:`.pth` file. See the documentation of the :mod:`site` module for more information. A slightly less convenient way is to edit the :file:`site.py` file in Python's standard library, and modify ``sys.path``. :file:`site.py` is automatically imported when the Python interpreter is executed, unless the :option:`-S` switch is supplied to suppress this behaviour. So you could simply edit :file:`site.py` and add two lines to it: .. code-block:: python import sys sys.path.append('/www/python/') However, if you reinstall the same major version of Python (perhaps when upgrading from 2.2 to 2.2.2, for example) :file:`site.py` will be overwritten by the stock version. You'd have to remember that it was modified and save a copy before doing the installation. There are two environment variables that can modify ``sys.path``. :envvar:`PYTHONHOME` sets an alternate value for the prefix of the Python installation. For example, if :envvar:`PYTHONHOME` is set to ``/www/python``, the search path will be set to ``['', '/www/python/lib/pythonX.Y/', '/www/python/lib/pythonX.Y/plat-linux2', ...]``. The :envvar:`PYTHONPATH` variable can be set to a list of paths that will be added to the beginning of ``sys.path``. For example, if :envvar:`PYTHONPATH` is set to ``/www/python:/opt/py``, the search path will begin with ``['/www/python', '/opt/py']``. (Note that directories must exist in order to be added to ``sys.path``; the :mod:`site` module removes paths that don't exist.) Finally, ``sys.path`` is just a regular Python list, so any Python application can modify it by adding or removing entries. .. _inst-config-files: Distutils Configuration Files ============================= As mentioned above, you can use Distutils configuration files to record personal or site preferences for any Distutils options. That is, any option to any command can be stored in one of two or three (depending on your platform) configuration files, which will be consulted before the command-line is parsed. This means that configuration files will override default values, and the command-line will in turn override configuration files. Furthermore, if multiple configuration files apply, values from "earlier" files are overridden by "later" files. .. _inst-config-filenames: Location and names of config files ---------------------------------- The names and locations of the configuration files vary slightly across platforms. On Unix and Mac OS X, the three configuration files (in the order they are processed) are: +--------------+----------------------------------------------------------+-------+ | Type of file | Location and filename | Notes | +==============+==========================================================+=======+ | system | :file:`{prefix}/lib/python{ver}/distutils/distutils.cfg` | \(1) | +--------------+----------------------------------------------------------+-------+ | personal | :file:`$HOME/.pydistutils.cfg` | \(2) | +--------------+----------------------------------------------------------+-------+ | local | :file:`setup.cfg` | \(3) | +--------------+----------------------------------------------------------+-------+ And on Windows, the configuration files are: +--------------+-------------------------------------------------+-------+ | Type of file | Location and filename | Notes | +==============+=================================================+=======+ | system | :file:`{prefix}\\Lib\\distutils\\distutils.cfg` | \(4) | +--------------+-------------------------------------------------+-------+ | personal | :file:`%HOME%\\pydistutils.cfg` | \(5) | +--------------+-------------------------------------------------+-------+ | local | :file:`setup.cfg` | \(3) | +--------------+-------------------------------------------------+-------+ On all platforms, the "personal" file can be temporarily disabled by passing the `--no-user-cfg` option. Notes: (1) Strictly speaking, the system-wide configuration file lives in the directory where the Distutils are installed; under Python 1.6 and later on Unix, this is as shown. For Python 1.5.2, the Distutils will normally be installed to :file:`{prefix}/lib/python1.5/site-packages/distutils`, so the system configuration file should be put there under Python 1.5.2. (2) On Unix, if the :envvar:`HOME` environment variable is not defined, the user's home directory will be determined with the :func:`getpwuid` function from the standard :mod:`pwd` module. This is done by the :func:`os.path.expanduser` function used by Distutils. (3) I.e., in the current directory (usually the location of the setup script). (4) (See also note (1).) Under Python 1.6 and later, Python's default "installation prefix" is :file:`C:\\Python`, so the system configuration file is normally :file:`C:\\Python\\Lib\\distutils\\distutils.cfg`. Under Python 1.5.2, the default prefix was :file:`C:\\Program Files\\Python`, and the Distutils were not part of the standard library---so the system configuration file would be :file:`C:\\Program Files\\Python\\distutils\\distutils.cfg` in a standard Python 1.5.2 installation under Windows. (5) On Windows, if the :envvar:`HOME` environment variable is not defined, :envvar:`USERPROFILE` then :envvar:`HOMEDRIVE` and :envvar:`HOMEPATH` will be tried. This is done by the :func:`os.path.expanduser` function used by Distutils. .. _inst-config-syntax: Syntax of config files ---------------------- The Distutils configuration files all have the same syntax. The config files are grouped into sections. There is one section for each Distutils command, plus a ``global`` section for global options that affect every command. Each section consists of one option per line, specified as ``option=value``. For example, the following is a complete config file that just forces all commands to run quietly by default: .. code-block:: ini [global] verbose=0 If this is installed as the system config file, it will affect all processing of any Python module distribution by any user on the current system. If it is installed as your personal config file (on systems that support them), it will affect only module distributions processed by you. And if it is used as the :file:`setup.cfg` for a particular module distribution, it affects only that distribution. You could override the default "build base" directory and make the :command:`build\*` commands always forcibly rebuild all files with the following: .. code-block:: ini [build] build-base=blib force=1 which corresponds to the command-line arguments :: python setup.py build --build-base=blib --force except that including the :command:`build` command on the command-line means that command will be run. Including a particular command in config files has no such implication; it only means that if the command is run, the options in the config file will apply. (Or if other commands that derive values from it are run, they will use the values in the config file.) You can find out the complete list of options for any command using the :option:`!--help` option, e.g.:: python setup.py build --help and you can find out the complete list of global options by using :option:`!--help` without a command:: python setup.py --help See also the "Reference" section of the "Distributing Python Modules" manual. .. _inst-building-ext: Building Extensions: Tips and Tricks ==================================== Whenever possible, the Distutils try to use the configuration information made available by the Python interpreter used to run the :file:`setup.py` script. For example, the same compiler and linker flags used to compile Python will also be used for compiling extensions. Usually this will work well, but in complicated situations this might be inappropriate. This section discusses how to override the usual Distutils behaviour. .. _inst-tweak-flags: Tweaking compiler/linker flags ------------------------------ Compiling a Python extension written in C or C++ will sometimes require specifying custom flags for the compiler and linker in order to use a particular library or produce a special kind of object code. This is especially true if the extension hasn't been tested on your platform, or if you're trying to cross-compile Python. In the most general case, the extension author might have foreseen that compiling the extensions would be complicated, and provided a :file:`Setup` file for you to edit. This will likely only be done if the module distribution contains many separate extension modules, or if they often require elaborate sets of compiler flags in order to work. A :file:`Setup` file, if present, is parsed in order to get a list of extensions to build. Each line in a :file:`Setup` describes a single module. Lines have the following structure:: module ... [sourcefile ...] [cpparg ...] [library ...] Let's examine each of the fields in turn. * *module* is the name of the extension module to be built, and should be a valid Python identifier. You can't just change this in order to rename a module (edits to the source code would also be needed), so this should be left alone. * *sourcefile* is anything that's likely to be a source code file, at least judging by the filename. Filenames ending in :file:`.c` are assumed to be written in C, filenames ending in :file:`.C`, :file:`.cc`, and :file:`.c++` are assumed to be C++, and filenames ending in :file:`.m` or :file:`.mm` are assumed to be in Objective C. * *cpparg* is an argument for the C preprocessor, and is anything starting with :option:`!-I`, :option:`!-D`, :option:`!-U` or :option:`!-C`. * *library* is anything ending in :file:`.a` or beginning with :option:`!-l` or :option:`!-L`. If a particular platform requires a special library on your platform, you can add it by editing the :file:`Setup` file and running ``python setup.py build``. For example, if the module defined by the line :: foo foomodule.c must be linked with the math library :file:`libm.a` on your platform, simply add :option:`!-lm` to the line:: foo foomodule.c -lm Arbitrary switches intended for the compiler or the linker can be supplied with the :option:`!-Xcompiler` *arg* and :option:`!-Xlinker` *arg* options:: foo foomodule.c -Xcompiler -o32 -Xlinker -shared -lm The next option after :option:`!-Xcompiler` and :option:`!-Xlinker` will be appended to the proper command line, so in the above example the compiler will be passed the :option:`!-o32` option, and the linker will be passed :option:`!-shared`. If a compiler option requires an argument, you'll have to supply multiple :option:`!-Xcompiler` options; for example, to pass ``-x c++`` the :file:`Setup` file would have to contain ``-Xcompiler -x -Xcompiler c++``. Compiler flags can also be supplied through setting the :envvar:`CFLAGS` environment variable. If set, the contents of :envvar:`CFLAGS` will be added to the compiler flags specified in the :file:`Setup` file. .. _inst-non-ms-compilers: Using non-Microsoft compilers on Windows ---------------------------------------- .. sectionauthor:: Rene Liebscher <R.Liebscher@gmx.de> Borland/CodeGear C++ ^^^^^^^^^^^^^^^^^^^^ This subsection describes the necessary steps to use Distutils with the Borland C++ compiler version 5.5. First you have to know that Borland's object file format (OMF) is different from the format used by the Python version you can download from the Python or ActiveState Web site. (Python is built with Microsoft Visual C++, which uses COFF as the object file format.) For this reason you have to convert Python's library :file:`python25.lib` into the Borland format. You can do this as follows: .. Should we mention that users have to create cfg-files for the compiler? .. see also http://community.borland.com/article/0,1410,21205,00.html :: coff2omf python25.lib python25_bcpp.lib The :file:`coff2omf` program comes with the Borland compiler. The file :file:`python25.lib` is in the :file:`Libs` directory of your Python installation. If your extension uses other libraries (zlib, ...) you have to convert them too. The converted files have to reside in the same directories as the normal libraries. How does Distutils manage to use these libraries with their changed names? If the extension needs a library (eg. :file:`foo`) Distutils checks first if it finds a library with suffix :file:`_bcpp` (eg. :file:`foo_bcpp.lib`) and then uses this library. In the case it doesn't find such a special library it uses the default name (:file:`foo.lib`.) [#]_ To let Distutils compile your extension with Borland C++ you now have to type:: python setup.py build --compiler=bcpp If you want to use the Borland C++ compiler as the default, you could specify this in your personal or system-wide configuration file for Distutils (see section :ref:`inst-config-files`.) .. seealso:: `C++Builder Compiler <https://www.embarcadero.com/products>`_ Information about the free C++ compiler from Borland, including links to the download pages. `Creating Python Extensions Using Borland's Free Compiler <http://www.cyberus.ca/~g_will/pyExtenDL.shtml>`_ Document describing how to use Borland's free command-line C++ compiler to build Python. GNU C / Cygwin / MinGW ^^^^^^^^^^^^^^^^^^^^^^ This section describes the necessary steps to use Distutils with the GNU C/C++ compilers in their Cygwin and MinGW distributions. [#]_ For a Python interpreter that was built with Cygwin, everything should work without any of these following steps. Not all extensions can be built with MinGW or Cygwin, but many can. Extensions most likely to not work are those that use C++ or depend on Microsoft Visual C extensions. To let Distutils compile your extension with Cygwin you have to type:: python setup.py build --compiler=cygwin and for Cygwin in no-cygwin mode [#]_ or for MinGW type:: python setup.py build --compiler=mingw32 If you want to use any of these options/compilers as default, you should consider writing it in your personal or system-wide configuration file for Distutils (see section :ref:`inst-config-files`.) Older Versions of Python and MinGW """""""""""""""""""""""""""""""""" The following instructions only apply if you're using a version of Python inferior to 2.4.1 with a MinGW inferior to 3.0.0 (with binutils-2.13.90-20030111-1). These compilers require some special libraries. This task is more complex than for Borland's C++, because there is no program to convert the library. First you have to create a list of symbols which the Python DLL exports. (You can find a good program for this task at https://sourceforge.net/projects/mingw/files/MinGW/Extension/pexports/). .. I don't understand what the next line means. --amk .. (inclusive the references on data structures.) :: pexports python25.dll >python25.def The location of an installed :file:`python25.dll` will depend on the installation options and the version and language of Windows. In a "just for me" installation, it will appear in the root of the installation directory. In a shared installation, it will be located in the system directory. Then you can create from these information an import library for gcc. :: /cygwin/bin/dlltool --dllname python25.dll --def python25.def --output-lib libpython25.a The resulting library has to be placed in the same directory as :file:`python25.lib`. (Should be the :file:`libs` directory under your Python installation directory.) If your extension uses other libraries (zlib,...) you might have to convert them too. The converted files have to reside in the same directories as the normal libraries do. .. seealso:: `Building Python modules on MS Windows platform with MinGW <http://old.zope.org/Members/als/tips/win32_mingw_modules>`_ Information about building the required libraries for the MinGW environment. .. rubric:: Footnotes .. [#] This also means you could replace all existing COFF-libraries with OMF-libraries of the same name. .. [#] Check https://www.sourceware.org/cygwin/ and http://www.mingw.org/ for more information .. [#] Then you have no POSIX emulation available, but you also don't need :file:`cygwin1.dll`. PK������� 3]8tW@)��@)�� ����������������reference/executionmodel.rst.txtnu�[��������PK������� 3]ioo����"������������)��reference/lexical_analysis.rst.txtnu�[��������PK������� 3],p&L�L�������������Ϯ��reference/expressions.rst.txtnu�[��������PK������� 3]ז _����������������h�reference/simple_stmts.rst.txtnu�[��������PK������� 3]@* �� ��%������������T�reference/toplevel_components.rst.txtnu�[��������PK������� 3]6 ������������������`�reference/grammar.rst.txtnu�[��������PK������� 3]z����������������a�reference/index.rst.txtnu�[��������PK������� 3] �� ��������������e�reference/import.rst.txtnu�[��������PK������� 3][��[��������������C�reference/introduction.rst.txtnu�[��������PK������� 3]bA'�'�������������/�reference/datamodel.rst.txtnu�[��������PK������� 3]F0q��q�� ������������^�reference/compound_stmts.rst.txtnu�[��������PK������� 3]JNW%��%��������������f�installing/index.rst.txtnu�[��������PK������� 3]6ȥ����������������i�using/unix.rst.txtnu�[��������PK������� 3]dCd��d��������������P�using/cmdline.rst.txtnu�[��������PK������� 3]9Y �����������������using/mac.rst.txtnu�[��������PK������� 3]J&��&��������������!�using/windows.rst.txtnu�[��������PK������� 3]fjlj����������������%�using/index.rst.txtnu�[��������PK������� 3]`b��b���������������extending/newtypes.rst.txtnu�[��������PK������� 3]:n �� ��������������8�extending/windows.rst.txtnu�[��������PK������� 3]vu����������������ER�extending/extending.rst.txtnu�[��������PK������� 3]@. ��. ��������������4�extending/index.rst.txtnu�[��������PK������� 3]o21��21�������������� @�extending/embedding.rst.txtnu�[��������PK������� 3]&D��D��#������������q�extending/newtypes_tutorial.rst.txtnu�[��������PK������� 3]c�����������������extending/building.rst.txtnu�[��������PK������� 3]qs&����������������O �license.rst.txtnu�[��������PK������� 3]-i����������������- �glossary.rst.txtnu�[��������PK������� 3]1���� ������������ �bugs.rst.txtnu�[��������PK������� 3]qA(��A(��������������> �distutils/examples.rst.txtnu�[��������PK������� 3]hQF���������������� �distutils/commandref.rst.txtnu�[��������PK������� 3]4'���������������� �distutils/configfile.rst.txtnu�[��������PK������� 3]UIu�u������������� �distutils/apiref.rst.txtnu�[��������PK������� 3]cG����������������o �distutils/extending.rst.txtnu�[��������PK������� 3]&6p���������������� �distutils/index.rst.txtnu�[��������PK������� 3]1W?@%��@%�������������� �distutils/sourcedist.rst.txtnu�[��������PK������� 3]7Ru������������������0 �distutils/uploading.rst.txtnu�[��������PK������� 3]R!��R!��������������H �distutils/introduction.rst.txtnu�[��������PK������� 3],z zX��zX�������������� �distutils/builtdist.rst.txtnu�[��������PK������� 3]#ήw��w��������������' �distutils/setupscript.rst.txtnu�[��������PK������� 3]o!��!�������������� �distutils/packageindex.rst.txtnu�[��������PK������� 3]���������������� �c-api/stable.rst.txtnu�[��������PK������� 3]Q %W)��W)�������������� �c-api/number.rst.txtnu�[��������PK������� 3]ڽЃ������������������x �c-api/set.rst.txtnu�[��������PK������� 3]nd ��d �������������� �c-api/slice.rst.txtnu�[��������PK������� 3]}����������������`�c-api/gen.rst.txtnu�[��������PK������� 3]@q�����������������c-api/utilities.rst.txtnu�[��������PK������� 3]FY �� ���������������c-api/marshal.rst.txtnu�[��������PK������� 3]A��A��������������*�c-api/veryhigh.rst.txtnu�[��������PK������� 3]<#��#��������������l�c-api/bytes.rst.txtnu�[��������PK������� 3]!k(��(��������������\�c-api/long.rst.txtnu�[��������PK������� 3]{V��V��������������b�c-api/buffer.rst.txtnu�[��������PK������� 3]"j��j���������������c-api/complex.rst.txtnu�[��������PK������� 3]8]R����������������P �c-api/bool.rst.txtnu�[��������PK������� 3]a!ZG��ZG��������������B%�c-api/module.rst.txtnu�[��������PK������� 3])ݖ����������������l�c-api/objbuffer.rst.txtnu�[��������PK������� 3]&��&��������������u�c-api/sys.rst.txtnu�[��������PK������� 3]#7m �� ��������������֜�c-api/method.rst.txtnu�[��������PK������� 3]׶Fƺ��ƺ���������������c-api/exceptions.rst.txtnu�[��������PK������� 3]u����������������*c�c-api/apiabiversion.rst.txtnu�[��������PK������� 3]ϤK ��K ��������������7l�c-api/weakref.rst.txtnu�[��������PK������� 3] TW] ��] ��������������v�c-api/refcounting.rst.txtnu�[��������PK������� 3]A҉����������������m�c-api/reflection.rst.txtnu�[��������PK������� 3]����������������>�c-api/concrete.rst.txtnu�[��������PK������� 3]fT�����������������c-api/init.rst.txtnu�[��������PK������� 3] |{��{��������������c]�c-api/type.rst.txtnu�[��������PK������� 3]6���������������� m�c-api/float.rst.txtnu�[��������PK������� 3]<k*h��h��������������&v�c-api/iterator.rst.txtnu�[��������PK������� 3]lm��m��������������|�c-api/capsule.rst.txtnu�[��������PK������� 3]^e��e���������������c-api/index.rst.txtnu�[��������PK������� 3]*_��_��������������.�c-api/tuple.rst.txtnu�[��������PK������� 3]n0HyS��yS��������������е�c-api/memory.rst.txtnu�[��������PK������� 3]Gw] ��] �������������� �c-api/allocation.rst.txtnu�[��������PK������� 3];>2��2��������������2�c-api/import.rst.txtnu�[��������PK������� 3]bj D�� D��������������F�c-api/object.rst.txtnu�[��������PK������� 3]u5 � �������������ߊ�c-api/unicode.rst.txtnu�[��������PK������� 3]%-œ�����������������c-api/sequence.rst.txtnu�[��������PK������� 3]�����������������c-api/list.rst.txtnu�[��������PK������� 3]1�����������������c-api/code.rst.txtnu�[��������PK������� 3]_����������������l�c-api/conversion.rst.txtnu�[��������PK������� 3]"0��0��������������@�c-api/objimpl.rst.txtnu�[��������PK������� 3] Op��Op���������������c-api/intro.rst.txtnu�[��������PK������� 3]S|aw��w��������������GP�c-api/arg.rst.txtnu�[��������PK������� 3]c����������������C�c-api/typeobj.rst.txtnu�[��������PK������� 3]M�����������������c-api/none.rst.txtnu�[��������PK������� 3](ޣ?����������������ƶ�c-api/datetime.rst.txtnu�[��������PK������� 3]N7 ��7 ���������������c-api/mapping.rst.txtnu�[��������PK������� 3]k4��4���������������c-api/coro.rst.txtnu�[��������PK������� 3]���������������� �c-api/abstract.rst.txtnu�[��������PK������� 3]5��5���������������c-api/structures.rst.txtnu�[��������PK������� 3]]%����������������A�c-api/gcsupport.rst.txtnu�[��������PK������� 3]D, ��, ��������������10�c-api/memoryview.rst.txtnu�[��������PK������� 3]60 �� ��������������9�c-api/file.rst.txtnu�[��������PK������� 3]q3����������������F�c-api/bytearray.rst.txtnu�[��������PK������� 3]aW��W��������������O�c-api/descriptor.rst.txtnu�[��������PK������� 3]mʼW����������������GT�c-api/cell.rst.txtnu�[��������PK������� 3]y^ �� ��������������\�c-api/function.rst.txtnu�[��������PK������� 3]K����������������wi�c-api/codec.rst.txtnu�[��������PK������� 3]nni��i��������������y|�c-api/iter.rst.txtnu�[��������PK������� 3]K?����������������$�c-api/dict.rst.txtnu�[��������PK������� 3]f�]��]�������������v�whatsnew/3.6.rst.txtnu�[��������PK������� 3].5�����������������whatsnew/3.0.rst.txtnu�[��������PK������� 3]U��U���������������whatsnew/3.1.rst.txtnu�[��������PK������� 3].W,�,��������������whatsnew/2.5.rst.txtnu�[��������PK������� 3]0'0�0�������������du�whatsnew/2.7.rst.txtnu�[��������PK������� 3]$wx��������������U�whatsnew/2.6.rst.txtnu�[��������PK������� 3]v��������������Y�whatsnew/3.3.rst.txtnu�[��������PK������� 3]#n���������������whatsnew/3.4.rst.txtnu�[��������PK������� 3]sc9"'��'��������������!�whatsnew/2.0.rst.txtnu�[��������PK������� 3]G8K���K��������������� v"�whatsnew/changelog.rst.txtnu�[��������PK������� 3]~e^c����������������v"�whatsnew/index.rst.txtnu�[��������PK������� 3]<F �� ��������������y"�whatsnew/2.4.rst.txtnu�[��������PK������� 3]Ԟڿ��������������4t#�whatsnew/3.2.rst.txtnu�[��������PK������� 3] ����������������7 %�whatsnew/2.2.rst.txtnu�[��������PK������� 3]xʷC^�C^������������� &�whatsnew/2.3.rst.txtnu�[��������PK������� 3]0Mn�n�������������j'�whatsnew/3.5.rst.txtnu�[��������PK������� 3]S_\v��v��������������(�whatsnew/2.1.rst.txtnu�[��������PK������� 3]oL����������������i)�contents.rst.txtnu�[��������PK������� 3]~7���� ������������k)�about.rst.txtnu�[��������PK������� 3]/dl ��l ��������������!r)�tutorial/interactive.rst.txtnu�[��������PK������� 3]x*#]��]��������������{)�tutorial/modules.rst.txtnu�[��������PK������� 3]n2U����������������)�tutorial/venv.rst.txtnu�[��������PK������� 3]&;��&;��������������)�tutorial/stdlib2.rst.txtnu�[��������PK������� 3]h �� ��������������-2*�tutorial/appendix.rst.txtnu�[��������PK������� 3]Fk3 �� ��������������D*�tutorial/whatnow.rst.txtnu�[��������PK������� 3]tȓh��h��������������jQ*�tutorial/controlflow.rst.txtnu�[��������PK������� 3]_��_��������������F*�tutorial/datastructures.rst.txtnu�[��������PK������� 3]rt:��:��������������+�tutorial/errors.rst.txtnu�[��������PK������� 3]2:R ��R ��������������T+�tutorial/index.rst.txtnu�[��������PK������� 3]{M*��M*��������������I^+�tutorial/stdlib.rst.txtnu�[��������PK������� 3]7z T��T��������������݈+�tutorial/classes.rst.txtnu�[��������PK������� 3] ����������������y,�tutorial/appetite.rst.txtnu�[��������PK������� 3]}@tD��D��������������`+,�tutorial/introduction.rst.txtnu�[��������PK������� 3]LY3C��C��������������Dp,�tutorial/inputoutput.rst.txtnu�[��������PK������� 3])!+��+��������������|,�tutorial/floatingpoint.rst.txtnu�[��������PK������� 3]v����������������,�tutorial/interpreter.rst.txtnu�[��������PK������� 3]RO��O��������������,�distributing/index.rst.txtnu�[��������PK������� 3],x@W��W��������������-�howto/pyporting.rst.txtnu�[��������PK������� 3]�c��c��������������m-�howto/curses.rst.txtnu�[��������PK������� 3]z+u����������������-�howto/functional.rst.txtnu�[��������PK������� 3]b-��-��������������.�howto/ipaddress.rst.txtnu�[��������PK������� 3]*H��H��������������.�howto/sockets.rst.txtnu�[��������PK������� 3]��������������a /�howto/logging-cookbook.rst.txtnu�[��������PK������� 3]gz7��7��������������0�howto/instrumentation.rst.txtnu�[��������PK������� 3]mH�����������������0�howto/index.rst.txtnu�[��������PK������� 3]]$]��$]��������������0�howto/argparse.rst.txtnu�[��������PK������� 3]# mY+��+��������������/1�howto/sorting.rst.txtnu�[��������PK������� 3] YT`��T`��������������[1�howto/urllib2.rst.txtnu�[��������PK������� 3]{��{��������������1�howto/unicode.rst.txtnu�[��������PK������� 3]cdƪ����������������82�howto/regex.rst.txtnu�[��������PK������� 3]ǭ����������������-3�howto/logging.rst.txtnu�[��������PK������� 3]%`����������������3�howto/cporting.rst.txtnu�[��������PK������� 3]Ӛ1����������������4�howto/clinic.rst.txtnu�[��������PK������� 3]roZA��A��������������5�howto/descriptor.rst.txtnu�[��������PK������� 3]Sw����������������J5�copyright.rst.txtnu�[��������PK������� 3]&fWv����������������L5�library/collections.rst.txtnu�[��������PK������� 3]?DS��S�������������� 6�library/xml.dom.rst.txtnu�[��������PK������� 3]; �� ��������������6�library/pipes.rst.txtnu�[��������PK������� 3]mh ��h ��������������6�library/linecache.rst.txtnu�[��������PK������� 3]Ӣ1*��*��������������6�library/reprlib.rst.txtnu�[��������PK������� 3]>i����!������������ 6�library/asyncio-eventloop.rst.txtnu�[��������PK������� 3]W5=2��=2��������������H7�library/collections.abc.rst.txtnu�[��������PK������� 3]"o����������������z7�library/distutils.rst.txtnu�[��������PK������� 3]TE5��E5��������������7�library/email.generator.rst.txtnu�[��������PK������� 3]ϖi��i�������������� 7�library/keyword.rst.txtnu�[��������PK������� 3]ŵad��d��������������к7�library/symtable.rst.txtnu�[��������PK������� 3]أ �� ��������������|7�library/undoc.rst.txtnu�[��������PK������� 3]c}@ �� ��������������7�library/constants.rst.txtnu�[��������PK������� 3]{ �� ��������������7�library/email.encoders.rst.txtnu�[��������PK������� 3]̓y"��y"��������������7�library/bz2.rst.txtnu�[��������PK������� 3].Mt��t�������������� 8�library/difflib.rst.txtnu�[��������PK������� 3] U����������������8�library/fcntl.rst.txtnu�[��������PK������� 3]駢-��-��������������ߝ8�library/email.mime.rst.txtnu�[��������PK������� 3]MeP��P��������������8�library/pydoc.rst.txtnu�[��������PK������� 3]$Ns����������������8�library/tabnanny.rst.txtnu�[��������PK������� 3]7����������������8�library/archiving.rst.txtnu�[��������PK������� 3]P.��.��������������8�library/readline.rst.txtnu�[��������PK������� 3]x{i��i��������������9�library/email.policy.rst.txtnu�[��������PK������� 3]R,/m'<��'<��������������p9�library/xml.sax.handler.rst.txtnu�[��������PK������� 3]CH������������������9�library/misc.rst.txtnu�[��������PK������� 3]uu����������������!9�library/hmac.rst.txtnu�[��������PK������� 3]D!'��'��������������9�library/tokenize.rst.txtnu�[��������PK������� 3]|��|��������������s9�library/tarfile.rst.txtnu�[��������PK������� 3]p٤����������������r:�library/xml.sax.rst.txtnu�[��������PK������� 3] p �� ��������������:�library/modulefinder.rst.txtnu�[��������PK������� 3]V����������������:�library/email.rst.txtnu�[��������PK������� 3]t=!6��!6��������������:�library/tempfile.rst.txtnu�[��������PK������� 3]�Q���������������� :�library/marshal.rst.txtnu�[��������PK������� 3]{Y����������������^�;�library/syslog.rst.txtnu�[��������PK������� 3]8����������������r;�library/bisect.rst.txtnu�[��������PK������� 3]I~BO�O�������������&;�library/unittest.mock.rst.txtnu�[��������PK������� 3]4!��!��������������w<�library/http.cookies.rst.txtnu�[��������PK������� 3]-T~u$�u$�������������<�library/decimal.rst.txtnu�[��������PK������� 3]d����������������Ͻ=�library/allos.rst.txtnu�[��������PK������� 3]˶I��I��������������=�library/http.client.rst.txtnu�[��������PK������� 3])k��k�������������� >�library/mailcap.rst.txtnu�[��������PK������� 3]վE��E��������������m>�library/traceback.rst.txtnu�[��������PK������� 3]]p����������������t_>�library/rlcompleter.rst.txtnu�[��������PK������� 3]?BC��BC��������������h>�library/signal.rst.txtnu�[��������PK������� 3],A����������������=>�library/errno.rst.txtnu�[��������PK������� 3]F9e/��e/��������������>�library/xml.sax.reader.rst.txtnu�[��������PK������� 3]RaTX"��"��������������>�library/selectors.rst.txtnu�[��������PK������� 3]!]E��E��������������?�library/ossaudiodev.rst.txtnu�[��������PK������� 3] (Am*�m*�������������_?�library/curses.rst.txtnu�[��������PK������� 3]?6Bc��c��������������@�library/modules.rst.txtnu�[��������PK������� 3]7=m��m��������������S@�library/functional.rst.txtnu�[��������PK������� 3]o<]��]�������������� @�library/zipfile.rst.txtnu�[��������PK������� 3]q$-����������������@�library/unix.rst.txtnu�[��������PK������� 3]UO����������������@�library/email.errors.rst.txtnu�[��������PK������� 3]fԭd��d��������������A�library/idle.rst.txtnu�[��������PK������� 3]ϕC����������������JfA�library/sndhdr.rst.txtnu�[��������PK������� 3]gA5�5�������������ZnA�library/turtle.rst.txtnu�[��������PK������� 3]nk����������������ՄB�library/debug.rst.txtnu�[��������PK������� 3]yH,'�,'�������������B�library/codecs.rst.txtnu�[��������PK������� 3]L2T~��T~��������������cC�library/ipaddress.rst.txtnu�[��������PK������� 3]7��7���������������-D�library/pprint.rst.txtnu�[��������PK������� 3]&!ǝ����������������eD�library/getopt.rst.txtnu�[��������PK������� 3]g -9��9��������������~D�library/custominterp.rst.txtnu�[��������PK������� 3]&����������������pD�library/termios.rst.txtnu�[��������PK������� 3]s!<��<��"������������^D�library/asyncio-subprocess.rst.txtnu�[��������PK������� 3]:GH��H��������������hD�library/text.rst.txtnu�[��������PK������� 3] ����������������D�library/plistlib.rst.txtnu�[��������PK������� 3]zOV4��V4��������������"D�library/asyncio-dev.rst.txtnu�[��������PK������� 3]̌o?�?�������������!E�library/doctest.rst.txtnu�[��������PK������� 3]S:pT����������������I:F�library/builtins.rst.txtnu�[��������PK������� 3]Lt����������������J@F�library/colorsys.rst.txtnu�[��������PK������� 3]6)��)��������������GF�library/textwrap.rst.txtnu�[��������PK������� 3]{��{�������������� qF�library/email.examples.rst.txtnu�[��������PK������� 3]_zX��zX��������������xF�library/cgi.rst.txtnu�[��������PK������� 3]U����������������F�library/xml.rst.txtnu�[��������PK������� 3]����������������F�library/asyncio-queue.rst.txtnu�[��������PK������� 3],-L.��.��������������F�library/abc.rst.txtnu�[��������PK������� 3][����������������")G�library/__main__.rst.txtnu�[��������PK������� 3]`J����������������,G�library/_thread.rst.txtnu�[��������PK������� 3]^J����������������HG�library/email.message.rst.txtnu�[��������PK������� 3]I~U��U��������������\G�library/atexit.rst.txtnu�[��������PK������� 3]D_~+)��)��������������G�library/audioop.rst.txtnu�[��������PK������� 3]7/l��l��������������H�library/json.rst.txtnu�[��������PK������� 3]����������������>oH�library/internet.rst.txtnu�[��������PK������� 3]����������������fsH�library/urllib.error.rst.txtnu�[��������PK������� 3]玴 �� ��������������I|H�library/glob.rst.txtnu�[��������PK������� 3]2\K=��=��������������AH�library/xml.sax.utils.rst.txtnu�[��������PK������� 3]ljS@��@��������������˙H�library/pickle.rst.txtnu�[��������PK������� 3]r����������������Q,I�library/tkinter.rst.txtnu�[��������PK������� 3]&2Z4��Z4��������������I�library/heapq.rst.txtnu�[��������PK������� 3]948"��8"��������������:I�library/uuid.rst.txtnu�[��������PK������� 3]����������������J�library/trace.rst.txtnu�[��������PK������� 3]Q����������������J�library/faulthandler.rst.txtnu�[��������PK������� 3](L/P��P��������������A8J�library/venv.rst.txtnu�[��������PK������� 3]g7aD��D��������������J�library/http.server.rst.txtnu�[��������PK������� 3] j �j �������������DJ�library/socket.rst.txtnu�[��������PK������� 3] ^ς����������������K�library/sys.rst.txtnu�[��������PK������� 3]U����������������ؼL�library/filesys.rst.txtnu�[��������PK������� 3]����������������L�library/html.rst.txtnu�[��������PK������� 3]R����������������<L�library/filecmp.rst.txtnu�[��������PK������� 3]]o����������������zL�library/nis.rst.txtnu�[��������PK������� 3]ud@Y��Y��������������L�library/threading.rst.txtnu�[��������PK������� 3]/(c��c��������������(~M�library/exceptions.rst.txtnu�[��������PK������� 3]'��'��������������TM�library/html.entities.rst.txtnu�[��������PK������� 3]=S��S��������������M�library/netdata.rst.txtnu�[��������PK������� 3]nCU-"��-"��������������bM�library/sysconfig.rst.txtnu�[��������PK������� 3] �� �������������� N�library/shelve.rst.txtnu�[��������PK������� 3]!u ����������������,N�library/binhex.rst.txtnu�[��������PK������� 3]2PS��PS��������������3N�library/weakref.rst.txtnu�[��������PK������� 3].*͛g��g��������������nN�library/hashlib.rst.txtnu�[��������PK������� 3]^85��85��������������PN�library/asyncore.rst.txtnu�[��������PK������� 3]?M}L��}L��������������$O�library/struct.rst.txtnu�[��������PK������� 3]7L �� ��������������qO�library/codeop.rst.txtnu�[��������PK������� 3]; c��c��������������}O�library/gettext.rst.txtnu�[��������PK������� 3]繁K:��:��������������O�library/xmlrpc.server.rst.txtnu�[��������PK������� 3]:W^����������������P�library/frameworks.rst.txtnu�[��������PK������� 3]Z_9 �� ��������������P�library/curses.panel.rst.txtnu�[��������PK������� 3]8]#��#��������������)P�library/email.util.rst.txtnu�[��������PK������� 3]����������������MP�library/urllib.rst.txtnu�[��������PK������� 3]_x"����������������PP�library/subprocess.rst.txtnu�[��������PK������� 3]}����������������0Q�library/windows.rst.txtnu�[��������PK������� 3]/\l��l��������������Q�library/http.cookiejar.rst.txtnu�[��������PK������� 3]g1����������������qQ�library/numeric.rst.txtnu�[��������PK������� 3]M%��%��������������tQ�library/site.rst.txtnu�[��������PK������� 3]y2��2��������������lQ�library/bdb.rst.txtnu�[��������PK������� 3],����������������hQ�library/xmlrpc.rst.txtnu�[��������PK������� 3](SAn��n��������������Q�library/select.rst.txtnu�[��������PK������� 3]׹(����������������=R�library/superseded.rst.txtnu�[��������PK������� 3]l6#��#��������������*?R�library/compileall.rst.txtnu�[��������PK������� 3][1/ ��/ ��������������bR�library/token.rst.txtnu�[��������PK������� 3]r=U��=U��������������lR�library/nntplib.rst.txtnu�[��������PK������� 3]t��t��������������{R�library/configparser.rst.txtnu�[��������PK������� 3]zޛ����������������;S�library/chunk.rst.txtnu�[��������PK������� 3]ǀ��ǀ��������������fS�library/time.rst.txtnu�[��������PK������� 3]B��B��"������������qT�library/concurrent.futures.rst.txtnu�[��������PK������� 3]a[P����������������XT�library/re.rst.txtnu�[��������PK������� 3]Ju����������������VU�library/copyreg.rst.txtnu�[��������PK������� 3]yf/��/��������������_U�library/sqlite3.rst.txtnu�[��������PK������� 3]Qg��g��������������U�library/tk.rst.txtnu�[��������PK������� 3]G_��_��%������������U�library/xml.etree.elementtree.rst.txtnu�[��������PK������� 3]xĮM$��M$��������������[V�library/email.charset.rst.txtnu�[��������PK������� 3]ᠽd��d��������������V�library/test.rst.txtnu�[��������PK������� 3]S����������������4W�library/numbers.rst.txtnu�[��������PK������� 3]dRT%��%��������������TW�library/gc.rst.txtnu�[��������PK������� 3]]0��0��������������zW�library/resource.rst.txtnu�[��������PK������� 3]k%;��;��������������(W�library/itertools.rst.txtnu�[��������PK������� 3]<X����������������AX�library/binascii.rst.txtnu�[��������PK������� 3]):k2��2��������������[X�library/timeit.rst.txtnu�[��������PK������� 3]:]u��]u��������������X�library/pathlib.rst.txtnu�[��������PK������� 3]8(�(�������������1Y�library/stdtypes.rst.txtnu�[��������PK������� 3]4M��M��������������[�library/csv.rst.txtnu�[��������PK������� 3]:gT��T��������������\�library/fpectl.rst.txtnu�[��������PK������� 3]>c��>c��������������<\�library/asyncio-task.rst.txtnu�[��������PK������� 3]SX6��X6��������������Ƃ\�library/zlib.rst.txtnu�[��������PK������� 3]@[��[��������������b\�library/smtplib.rst.txtnu�[��������PK������� 3],���� ������������8]�library/logging.handlers.rst.txtnu�[��������PK������� 3] �C��C��������������]�library/lzma.rst.txtnu�[��������PK������� 3] Ղ��Ղ��&������������]�library/email.compat32-message.rst.txtnu�[��������PK������� 3]<";��;��������������ɂ^�library/statistics.rst.txtnu�[��������PK������� 3]_����������������ؾ^�library/datatypes.rst.txtnu�[��������PK������� 3](=����������������^�library/mm.rst.txtnu�[��������PK������� 3]\����������������^�library/index.rst.txtnu�[��������PK������� 3] ����������������^�library/enum.rst.txtnu�[��������PK������� 3]Iu*��*��������������KM_�library/calendar.rst.txtnu�[��������PK������� 3]*l.�.�������������x_�library/argparse.rst.txtnu�[��������PK������� 3],',�,�������������o`�library/optparse.rst.txtnu�[��������PK������� 3]'����������������{a�library/development.rst.txtnu�[��������PK������� 3]'= ^�� ^��������������a�library/winreg.rst.txtnu�[��������PK������� 3]#}��}��������������5b�library/__future__.rst.txtnu�[��������PK������� 3]M~��~��&������������Jb�library/unittest.mock-examples.rst.txtnu�[��������PK������� 3]r<n��n��������������pb�library/posix.rst.txtnu�[��������PK������� 3]Yj r#��#��������������#c�library/email.header.rst.txtnu�[��������PK������� 3]ɟ<��<��������������S2c�library/imp.rst.txtnu�[��������PK������� 3]�?d>��>��������������Roc�library/2to3.rst.txtnu�[��������PK������� 3]m����������������c�library/markup.rst.txtnu�[��������PK������� 3]aKM&��M&��������������c�library/stat.rst.txtnu�[��������PK������� 3]4C����������������.c�library/distribution.rst.txtnu�[��������PK������� 3]_$%F��%F��$������������>c�library/email.headerregistry.rst.txtnu�[��������PK������� 3]N[n��n��������������d�library/contextlib.rst.txtnu�[��������PK������� 3] 6;��;��������������d�library/importlib.rst.txtnu�[��������PK������� 3]o+~c��~c��������������_e�library/locale.rst.txtnu�[��������PK������� 3]C����������������Xe�library/python.rst.txtnu�[��������PK������� 3]%1(��(��������������ye�library/msvcrt.rst.txtnu�[��������PK������� 3]ؔN��N��������������e�library/warnings.rst.txtnu�[��������PK������� 3]h+.7��7��������������%f�library/dis.rst.txtnu�[��������PK������� 3]#����������������af�library/i18n.rst.txtnu�[��������PK������� 3]G.��.��������������=f�library/string.rst.txtnu�[��������PK������� 3]9Ǻ\��\��������������1g�library/socketserver.rst.txtnu�[��������PK������� 3] n����������������g�library/pickletools.rst.txtnu�[��������PK������� 3]+��+��������������yg�library/mmap.rst.txtnu�[��������PK������� 3]3i����������������g�library/ipc.rst.txtnu�[��������PK������� 3]Ϋ������������������ig�library/concurrent.rst.txtnu�[��������PK������� 3]Ln$��n$��������������^g�library/cmath.rst.txtnu�[��������PK������� 3]|&e��e��������������g�library/shutil.rst.txtnu�[��������PK������� 3]& �� ��������������Wh�library/spwd.rst.txtnu�[��������PK������� 3]/q ��q ��������������ch�library/grp.rst.txtnu�[��������PK������� 3]]Uo 6*��6*��������������{mh�library/smtpd.rst.txtnu�[��������PK������� 3]P(hij �� ��������������h�library/pwd.rst.txtnu�[��������PK������� 3]QU!����������������h�library/wave.rst.txtnu�[��������PK������� 3]-O��O��������������h�library/aifc.rst.txtnu�[��������PK������� 3]$B����������������h�library/dummy_threading.rst.txtnu�[��������PK������� 3]CI��I��������������h�library/tty.rst.txtnu�[��������PK������� 3]_:f7��7��������������h�library/dbm.rst.txtnu�[��������PK������� 3]]����������������i�library/sched.rst.txtnu�[��������PK������� 3]&'��'��������������.i�library/types.rst.txtnu�[��������PK������� 3]Ni-v#��v#��������������Vi�library/asyncio-sync.rst.txtnu�[��������PK������� 3]jR>�>�������������yi�library/os.rst.txtnu�[��������PK������� 3]l\8��8��������������ak�library/asyncio-stream.rst.txtnu�[��������PK������� 3]4dl����������������k�library/macpath.rst.txtnu�[��������PK������� 3]gVD^��^��������������k�library/logging.config.rst.txtnu�[��������PK������� 3].5<*��<*��������������Sl�library/array.rst.txtnu�[��������PK������� 3]|vDO��O��������������~l�library/persistence.rst.txtnu�[��������PK������� 3]=}X��}X��������������l�library/tkinter.tix.rst.txtnu�[��������PK������� 3]pk] ��] ��������������cl�library/runpy.rst.txtnu�[��������PK������� 3]ѽ �� ��������������l�library/pyclbr.rst.txtnu�[��������PK������� 3]c .�.�������������,m�library/multiprocessing.rst.txtnu�[��������PK������� 3])=99@��9@��������������n�library/shlex.rst.txtnu�[��������PK������� 3]So��o��������������'n�library/pyexpat.rst.txtnu�[��������PK������� 3])M5��5��������������:Po�library/cmd.rst.txtnu�[��������PK������� 3]~����������������`o�library/code.rst.txtnu�[��������PK������� 3]@aKK��K��������������-o�library/pdb.rst.txtnu�[��������PK������� 3]רVP!��P!��������������no�library/asynchat.rst.txtnu�[��������PK������� 3]h ����������������p�library/symbol.rst.txtnu�[��������PK������� 3]FRb��Rb�� ������������p�library/asyncio-protocol.rst.txtnu�[��������PK������� 3]W+#@��#@��������������yp�library/os.path.rst.txtnu�[��������PK������� 3]F+*��*��������������'p�library/secrets.rst.txtnu�[��������PK������� 3]Dz ��z ��������������p�library/imghdr.rst.txtnu�[��������PK������� 3]| -J��-J��������������Xp�library/operator.rst.txtnu�[��������PK������� 3]0!AC��C��������������'q�library/zipapp.rst.txtnu�[��������PK������� 3]Uˁ'��'��������������lq�library/xml.dom.minidom.rst.txtnu�[��������PK������� 3]EH0O��O��������������(q�library/imaplib.rst.txtnu�[��������PK������� 3]nw����������������q�library/asyncio.rst.txtnu�[��������PK������� 3]r����������������%q�library/wsgiref.rst.txtnu�[��������PK������� 3]m U ��U ��������������nr�library/intro.rst.txtnu�[��������PK������� 3]Zsf ��f ��������������yr�library/cgitb.rst.txtnu�[��������PK������� 3]&(o&��o&��������������Ȅr�library/mimetypes.rst.txtnu�[��������PK������� 3]gxl�xl�������������r�library/unittest.rst.txtnu�[��������PK������� 3]^%Z�%Z�������������@t�library/ctypes.rst.txtnu�[��������PK������� 3]JM'��'��������������ru�library/ast.rst.txtnu�[��������PK������� 3]XO��O��������������u�library/mailbox.rst.txtnu�[��������PK������� 3]W"����������������3v�library/sunau.rst.txtnu�[��������PK������� 3]& ����������������Vv�library/crypt.rst.txtnu�[��������PK������� 3]0Z��Z��������������v�library/getpass.rst.txtnu�[��������PK������� 3]RTOGH��GH��������������>v�library/functools.rst.txtnu�[��������PK������� 3]}d����������������w�library/http.rst.txtnu�[��������PK������� 3]qH��H��������������.w�library/msilib.rst.txtnu�[��������PK������� 3]B��B��������������ww�library/xml.dom.pulldom.rst.txtnu�[��������PK������� 3] 8m ��m ��������������Uw�library/gzip.rst.txtnu�[��������PK������� 3]<,y��y��������������w�library/tkinter.ttk.rst.txtnu�[��������PK������� 3](q^'8��'8��������������ʑx�library/math.rst.txtnu�[��������PK������� 3]w{ڜa�a�������������5x�library/datetime.rst.txtnu�[��������PK������� 3]5w�w�������������,z�library/ssl.rst.txtnu�[��������PK������� 3]łW����������������q{�library/zipimport.rst.txtnu�[��������PK������� 3]s��s��������������g{�library/fileinput.rst.txtnu�[��������PK������� 3]uAm��Am��������������#{�library/profile.rst.txtnu�[��������PK������� 3]O �� ��������������G|�library/netrc.rst.txtnu�[��������PK������� 3]{/U(��(��������������S|�library/base64.rst.txtnu�[��������PK������� 3]H&w����������������x||�library/inspect.rst.txtnu�[��������PK������� 3]]D��D��������������gO}�library/ftplib.rst.txtnu�[��������PK������� 3]Gizʺ3��3��������������=}�library/formatter.rst.txtnu�[��������PK������� 3].%}d��d��������������@}�library/queue.rst.txtnu�[��������PK������� 3]S �� ��������������}�library/quopri.rst.txtnu�[��������PK������� 3]V_p����������������:}�library/typing.rst.txtnu�[��������PK������� 3]u. ,�� ,��������������u~�library/html.parser.rst.txtnu�[��������PK������� 3]W$V����������������~�library/stringprep.rst.txtnu�[��������PK������� 3]87&��7&��������������~�library/webbrowser.rst.txtnu�[��������PK������� 3] ?��?��������������t~�library/logging.rst.txtnu�[��������PK������� 3]Xܴގ�����������������library/xdrlib.rst.txtnu�[��������PK������� 3]aE 7�� 7���������������library/email.parser.rst.txtnu�[��������PK������� 3]:fЎ����������������&�library/binary.rst.txtnu�[��������PK������� 3]߭e����"�������������library/asyncio-eventloops.rst.txtnu�[��������PK������� 3]3 ��3 ��������������8>�library/pty.rst.txtnu�[��������PK������� 3]eh����������������J�library/fileformats.rst.txtnu�[��������PK������� 3]҅m,;��,;��������������L�library/parser.rst.txtnu�[��������PK������� 3]s9 �� ���������������library/email.iterators.rst.txtnu�[��������PK������� 3]2����������������Ē�library/fractions.rst.txtnu�[��������PK������� 3]}t����������������髀�library/concurrency.rst.txtnu�[��������PK������� 3]?e �� ��������������ծ�library/fnmatch.rst.txtnu�[��������PK������� 3]f��������������;�library/functions.rst.txtnu�[��������PK������� 3]\C �� ��������������IӁ�library/py_compile.rst.txtnu�[��������PK������� 3]$2 ��2 ���������������library/copy.rst.txtnu�[��������PK������� 3]Z"�����������������library/unicodedata.rst.txtnu�[��������PK������� 3]S �� ���������������library/poplib.rst.txtnu�[��������PK������� 3]:����������������,'�library/winsound.rst.txtnu�[��������PK������� 3]@����������������;�library/io.rst.txtnu�[��������PK������� 3]!c����������������Lق�library/urllib.request.rst.txtnu�[��������PK������� 3].�MO"��"��������������z�library/curses.ascii.rst.txtnu�[��������PK������� 3]u �� ���������������library/othergui.rst.txtnu�[��������PK������� 3]_AԘ#��#��$�������������library/email.contentmanager.rst.txtnu�[��������PK������� 3]{mØ%��%���������������library/platform.rst.txtnu�[��������PK������� 3]͍K"��"��������������9�library/pkgutil.rst.txtnu�[��������PK������� 3]P �� ��"������������\�library/urllib.robotparser.rst.txtnu�[��������PK������� 3]h1GH��GH��������������h�library/random.rst.txtnu�[��������PK������� 3]7{��{���������������library/ensurepip.rst.txtnu�[��������PK������� 3]%����������������UĄ�library/telnetlib.rst.txtnu�[��������PK������� 3]S����������������y�library/crypto.rst.txtnu�[��������PK������� 3]2˲����������������Z�library/_dummy_thread.rst.txtnu�[��������PK������� 3]w �� ���������������library/language.rst.txtnu�[��������PK������� 3]S ��S ���������������library/uu.rst.txtnu�[��������PK������� 3]#^qj��j���������������library/urllib.parse.rst.txtnu�[��������PK������� 3]y����$������������_�library/tkinter.scrolledtext.rst.txtnu�[��������PK������� 3]NIV��IV��������������d�library/tracemalloc.rst.txtnu�[��������PK������� 3]7cZ��Z��������������g�library/xmlrpc.client.rst.txtnu�[��������PK������� 3]gP��P���������������faq/general.rst.txtnu�[��������PK������� 3]ioa��a��������������g�faq/design.rst.txtnu�[��������PK������� 3]H�����������������faq/gui.rst.txtnu�[��������PK������� 3]Lz��Lz���������������faq/library.rst.txtnu�[��������PK������� 3]Y;- �� ���������������faq/installed.rst.txtnu�[��������PK������� 3]x]<��]<���������������faq/windows.rst.txtnu�[��������PK������� 3]#O\=��\=��������������LJ�faq/extending.rst.txtnu�[��������PK������� 3] Xa�a�������������0�faq/programming.rst.txtnu�[��������PK������� 3]Vv���������������� �faq/index.rst.txtnu�[��������PK������� 3]2x����������������/�install/index.rst.txtnu�[��������PK������E҉���