mercurial: implement a source transforming module loader on Python 3 The most painful part of ensuring Python code runs on both Python 2 and 3 is string encoding. Making this difficult is that string literals in Python 2 are bytes and string literals in Python 3 are unicode. So, to ensure consistent types are used, you have to use "from __future__ import unicode_literals" and/or prefix literals with their type (e.g. b'foo' or u'foo'). Nearly every string in Mercurial is bytes. So, to use the same source code on both Python 2 and 3 would require prefixing nearly every string literal with "b" to make it a byte literal. This is ugly and not something mpm is willing to do at this point in time. This patch implements a custom module loader on Python 3 that performs source transformation to convert string literals (unicode in Python 3) to byte literals. In effect, it changes Python 3's string literals to behave like Python 2's. In addition, the module loader recognizes well-known built-in functions (getattr, setattr, hasattr) and methods (encode and decode) that barf when bytes are used and prevents these from being rewritten. This prevents excessive source changes to accommodate this change (we would have to rewrite every occurrence of these functions passing string literals otherwise). The module loader is only used on Python packages belonging to Mercurial. The loader works by tokenizing the loaded source and replacing "string" tokens if necessary. The modified token stream is untokenized back to source and loaded like normal. This does add some overhead. However, this all occurs before caching: .pyc files will cache the transformed version. This means the transformation penalty is only paid on first load. As the extensive inline comments explain, the presence of a custom source transformer invalidates assumptions made by Python's built-in bytecode caching mechanism. So, we have to wrap bytecode loading and writing and add an additional header to bytecode files to facilitate additional cache validation when the source transformations change in the future. There are still a few things this code doesn't handle well, namely support for zip files as module sources and for extensions. Since Mercurial doesn't officially support Python 3 yet, I'm inclined to leave these as to-do items: getting a basic module loading mechanism in place to unblock further Python 3 porting effort is more important than comprehensive module importing support. check-py3-compat.py has been updated to ignore frames. This is necessary because CPython has built-in code to strip frames from the built-in importer. When our custom code is present, this doesn't work and the frames get all messed up. The new code is not perfect. It works for now. But once you start chasing import failures you find some edge cases where the files aren't being printed properly. This only burdens people doing future Python 3 porting work so I'm inclined to punt on the issue: the most important thing is for the source transforming module loader to land. There was a bit of churn in test-check-py3-compat.t because we now trip up on str/unicode/bytes failures as a result of source transformation. This is unfortunate but what are you going to do. It's worth noting that other approaches were investigated. We considered using a custom file encoding whose decode() would apply source transformations. This was rejected because it would require each source file to declare its custom Mercurial encoding. Furthermore, when changing the source transformation we'd need to version bump the encoding name otherwise the module caching layer wouldn't know the .pyc file was invalidated. This would mean mass updating every file when the source transformation changes. Yuck. We also considered transforming at the AST layer. However, Python's ast module is quite gnarly and doing AST transforms is quite complicated, even for trivial rewrites. There are whole Python packages that exist to make AST transformations usable. AST transforms would still require import machinery, so the choice was basically to perform source-level, token-level, or ast-level transforms. Token-level rewriting delivers the metadata we need to rewrite intelligently while being relatively easy to understand. So it won. General consensus seems to be that this approach is the best available to avoid bulk rewriting of '' to b''. However, we aren't confident that this approach will never be a future maintenance burden. This approach does unblock serious Python 3 porting efforts. So we can re-evaulate once more work is done to support Python 3.

     1

#!/usr/bin/env python

     2

#

     3

# check-py3-compat - check Python 3 compatibility of Mercurial files

     4

#

     5

# Copyright 2015 Gregory Szorc <gregory.szorc@gmail.com>

     6

#

     7

# This software may be used and distributed according to the terms of the

     8

# GNU General Public License version 2 or any later version.

     9

    10

from __future__ import absolute_import, print_function

    11

    12

import ast

    13

import imp

    14

import os

    15

import sys

    16

import traceback

    17

    18

def check_compat_py2(f):

    19

    """Check Python 3 compatibility for a file with Python 2"""

    20

    with open(f, 'rb') as fh:

    21

        content = fh.read()

    22

    root = ast.parse(content)

    23

    24

    # Ignore empty files.

    25

    if not root.body:

    26

        return

    27

    28

    futures = set()

    29

    haveprint = False

    30

    for node in ast.walk(root):

    31

        if isinstance(node, ast.ImportFrom):

    32

            if node.module == '__future__':

    33

                futures |= set(n.name for n in node.names)

    34

        elif isinstance(node, ast.Print):

    35

            haveprint = True

    36

    37

    if 'absolute_import' not in futures:

    38

        print('%s not using absolute_import' % f)

    39

    if haveprint and 'print_function' not in futures:

    40

        print('%s requires print_function' % f)

    41

    42

def check_compat_py3(f):

    43

    """Check Python 3 compatibility of a file with Python 3."""

    44

    with open(f, 'rb') as fh:

    45

        content = fh.read()

    46

    47

    try:

    48

        ast.parse(content)

    49

    except SyntaxError as e:

    50

        print('%s: invalid syntax: %s' % (f, e))

    51

        return

    52

    53

    # Try to import the module.

    54

    # For now we only support mercurial.* and hgext.* modules because figuring

    55

    # out module paths for things not in a package can be confusing.

    56

    if f.startswith(('hgext/', 'mercurial/')) and not f.endswith('__init__.py'):

    57

        assert f.endswith('.py')

    58

        name = f.replace('/', '.')[:-3]

    59

        with open(f, 'r') as fh:

    60

            try:

    61

                imp.load_module(name, fh, '', ('py', 'r', imp.PY_SOURCE))

    62

            except Exception as e:

    63

                exc_type, exc_value, tb = sys.exc_info()

    64

                # We walk the stack and ignore frames from our custom importer,

    65

                # import mechanisms, and stdlib modules. This kinda/sorta

    66

                # emulates CPython behavior in import.c while also attempting

    67

                # to pin blame on a Mercurial file.

    68

                for frame in reversed(traceback.extract_tb(tb)):

    69

                    if frame.name == '_call_with_frames_removed':

    70

                        continue

    71

                    if 'importlib' in frame.filename:

    72

                        continue

    73

                    if 'mercurial/__init__.py' in frame.filename:

    74

                        continue

    75

                    if frame.filename.startswith(sys.prefix):

    76

                        continue

    77

                    break

    78

    79

                if frame.filename:

    80

                    filename = os.path.basename(frame.filename)

    81

                    print('%s: error importing: <%s> %s (error at %s:%d)' % (

    82

                          f, type(e).__name__, e, filename, frame.lineno))

    83

                else:

    84

                    print('%s: error importing module: <%s> %s (line %d)' % (

    85

                          f, type(e).__name__, e, frame.lineno))

    86

    87

if __name__ == '__main__':

    88

    if sys.version_info[0] == 2:

    89

        fn = check_compat_py2

    90

    else:

    91

        fn = check_compat_py3

    92

    93

    for f in sys.argv[1:]:

    94

        fn(f)

    95

    96

    sys.exit(0)