hgdemandimport/demandimportpy3.py
author Gregory Szorc <gregory.szorc@gmail.com>
Sat, 02 Nov 2019 11:42:46 -0700
branchstable
changeset 43378 2d31ef3fb494
parent 43076 2372284d9457
child 44117 c5e0a9b97b8a
permissions -rw-r--r--
demandimportpy3: only use lazy extension loader on Python 3.6+ There was an inline comment denoting a bug in the lazy extension loader on Python 3.5 which prevents it from working there. But the code was not conditional on the Python version. The result of this was a myriad of failures on Python 3.5 due to getattr() and friends not working on lazy extension modules. By making extension modules non-lazy on Python 3.5, we reduce the number of test failures from 48 to 22 on that Python version.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
     1
# demandimportpy3 - global demand-loading of modules for Mercurial
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
     2
#
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
     3
# Copyright 2017 Facebook Inc.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
     4
#
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
     5
# This software may be used and distributed according to the terms of the
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
     6
# GNU General Public License version 2 or any later version.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
     7
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
     8
"""Lazy loading for Python 3.6 and above.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
     9
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    10
This uses the new importlib finder/loader functionality available in Python 3.5
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    11
and up. The code reuses most of the mechanics implemented inside importlib.util,
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    12
but with a few additions:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    13
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    14
* Allow excluding certain modules from lazy imports.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    15
* Expose an interface that's substantially the same as demandimport for
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    16
  Python 2.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    17
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    18
This also has some limitations compared to the Python 2 implementation:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    19
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    20
* Much of the logic is per-package, not per-module, so any packages loaded
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    21
  before demandimport is enabled will not be lazily imported in the future. In
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    22
  practice, we only expect builtins to be loaded before demandimport is
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    23
  enabled.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    24
"""
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    25
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    26
# This line is unnecessary, but it satisfies test-check-py3-compat.t.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    27
from __future__ import absolute_import
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    28
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    29
import contextlib
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    30
import importlib.abc
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    31
import importlib.machinery
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    32
import importlib.util
33898
3595e4e0ae57 demandimportpy3: update to pass import checker
Augie Fackler <raf@durin42.com>
parents: 33859
diff changeset
    33
import sys
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    34
42474
adb636392b3f demandimport: add tracing coverage for Python 3
Augie Fackler <augie@google.com>
parents: 37843
diff changeset
    35
from . import tracing
adb636392b3f demandimport: add tracing coverage for Python 3
Augie Fackler <augie@google.com>
parents: 37843
diff changeset
    36
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    37
_deactivated = False
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    38
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    39
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    40
class _lazyloaderex(importlib.util.LazyLoader):
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    41
    """This is a LazyLoader except it also follows the _deactivated global and
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    42
    the ignore list.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    43
    """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    44
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    45
    def exec_module(self, module):
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    46
        """Make the module load lazily."""
42474
adb636392b3f demandimport: add tracing coverage for Python 3
Augie Fackler <augie@google.com>
parents: 37843
diff changeset
    47
        with tracing.log('demandimport %s', module):
adb636392b3f demandimport: add tracing coverage for Python 3
Augie Fackler <augie@google.com>
parents: 37843
diff changeset
    48
            if _deactivated or module.__name__ in ignores:
adb636392b3f demandimport: add tracing coverage for Python 3
Augie Fackler <augie@google.com>
parents: 37843
diff changeset
    49
                self.loader.exec_module(module)
adb636392b3f demandimport: add tracing coverage for Python 3
Augie Fackler <augie@google.com>
parents: 37843
diff changeset
    50
            else:
adb636392b3f demandimport: add tracing coverage for Python 3
Augie Fackler <augie@google.com>
parents: 37843
diff changeset
    51
                super().exec_module(module)
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    52
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    53
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    54
# This is 3.6+ because with Python 3.5 it isn't possible to lazily load
35524
fcb1ecf2bef7 hgdemandimport: use correct hyperlink to python-bug in comments (issue5765)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 33898
diff changeset
    55
# extensions. See the discussion in https://bugs.python.org/issue26186 for more.
43378
2d31ef3fb494 demandimportpy3: only use lazy extension loader on Python 3.6+
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43076
diff changeset
    56
if sys.version_info[0:2] >= (3, 6):
2d31ef3fb494 demandimportpy3: only use lazy extension loader on Python 3.6+
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43076
diff changeset
    57
    _extensions_loader = _lazyloaderex.factory(
2d31ef3fb494 demandimportpy3: only use lazy extension loader on Python 3.6+
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43076
diff changeset
    58
        importlib.machinery.ExtensionFileLoader
2d31ef3fb494 demandimportpy3: only use lazy extension loader on Python 3.6+
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43076
diff changeset
    59
    )
2d31ef3fb494 demandimportpy3: only use lazy extension loader on Python 3.6+
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43076
diff changeset
    60
else:
2d31ef3fb494 demandimportpy3: only use lazy extension loader on Python 3.6+
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43076
diff changeset
    61
    _extensions_loader = importlib.machinery.ExtensionFileLoader
2d31ef3fb494 demandimportpy3: only use lazy extension loader on Python 3.6+
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43076
diff changeset
    62
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    63
_bytecode_loader = _lazyloaderex.factory(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    64
    importlib.machinery.SourcelessFileLoader
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    65
)
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    66
_source_loader = _lazyloaderex.factory(importlib.machinery.SourceFileLoader)
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    67
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    68
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    69
def _makefinder(path):
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    70
    return importlib.machinery.FileFinder(
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    71
        path,
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    72
        # This is the order in which loaders are passed in in core Python.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    73
        (_extensions_loader, importlib.machinery.EXTENSION_SUFFIXES),
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    74
        (_source_loader, importlib.machinery.SOURCE_SUFFIXES),
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    75
        (_bytecode_loader, importlib.machinery.BYTECODE_SUFFIXES),
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    76
    )
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    77
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    78
37843
670eb4fa1b86 demandimport: make module ignores a set (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35524
diff changeset
    79
ignores = set()
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    80
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    81
37843
670eb4fa1b86 demandimport: make module ignores a set (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35524
diff changeset
    82
def init(ignoreset):
670eb4fa1b86 demandimport: make module ignores a set (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35524
diff changeset
    83
    global ignores
670eb4fa1b86 demandimport: make module ignores a set (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35524
diff changeset
    84
    ignores = ignoreset
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    85
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    86
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    87
def isenabled():
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    88
    return _makefinder in sys.path_hooks and not _deactivated
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    89
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    90
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    91
def disable():
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    92
    try:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    93
        while True:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    94
            sys.path_hooks.remove(_makefinder)
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    95
    except ValueError:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    96
        pass
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    97
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    98
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    99
def enable():
33859
8fb5212652ec demandimport: move HGDEMANDIMPORT test to __init__.py
Jun Wu <quark@fb.com>
parents: 32423
diff changeset
   100
    sys.path_hooks.insert(0, _makefinder)
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   101
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
   102
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   103
@contextlib.contextmanager
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   104
def deactivated():
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   105
    # This implementation is a bit different from Python 2's. Python 3
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   106
    # maintains a per-package finder cache in sys.path_importer_cache (see
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   107
    # PEP 302). This means that we can't just call disable + enable.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   108
    # If we do that, in situations like:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   109
    #
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   110
    #   demandimport.enable()
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   111
    #   ...
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   112
    #   from foo.bar import mod1
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   113
    #   with demandimport.deactivated():
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   114
    #       from foo.bar import mod2
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   115
    #
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   116
    # mod2 will be imported lazily. (The converse also holds -- whatever finder
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   117
    # first gets cached will be used.)
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   118
    #
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   119
    # Instead, have a global flag the LazyLoader can use.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   120
    global _deactivated
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   121
    demandenabled = isenabled()
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   122
    if demandenabled:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   123
        _deactivated = True
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   124
    try:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   125
        yield
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   126
    finally:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   127
        if demandenabled:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   128
            _deactivated = False