hgdemandimport/demandimportpy3.py
author Gregory Szorc <gregory.szorc@gmail.com>
Mon, 20 Jan 2020 23:42:19 -0800
changeset 44117 c5e0a9b97b8a
parent 43378 2d31ef3fb494
child 44118 f81c17ec303c
permissions -rw-r--r--
hgdemandimport: disable on Python 3.5 The demand importer functionality isn't working at all on Python 3.5. I'm not sure what's wrong. Since it isn't working, let's disable it completely. ``` $ HGRCPATH= hyperfine -w 1 -r 50 -- "~/.pyenv/versions/3.5.9/bin/python ./hg version" \ "HGDEMANDIMPORT=disable ~/.pyenv/versions/3.5.9/bin/python ./hg version" Benchmark #1: ~/.pyenv/versions/3.5.9/bin/python ./hg version Time (mean ± σ): 163.7 ms ± 2.2 ms [User: 148.5 ms, System: 15.7 ms] Range (min … max): 161.0 ms … 170.2 ms 50 runs Benchmark #2: HGDEMANDIMPORT=disable ~/.pyenv/versions/3.5.9/bin/python ./hg version Time (mean ± σ): 164.3 ms ± 1.4 ms [User: 148.2 ms, System: 16.6 ms] Range (min … max): 161.4 ms … 169.8 ms 50 runs ``` Differential Revision: https://phab.mercurial-scm.org/D7953
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
     1
# demandimportpy3 - global demand-loading of modules for Mercurial
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
     2
#
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
     3
# Copyright 2017 Facebook Inc.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
     4
#
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
     5
# This software may be used and distributed according to the terms of the
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
     6
# GNU General Public License version 2 or any later version.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
     7
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
     8
"""Lazy loading for Python 3.6 and above.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
     9
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    10
This uses the new importlib finder/loader functionality available in Python 3.5
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    11
and up. The code reuses most of the mechanics implemented inside importlib.util,
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    12
but with a few additions:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    13
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    14
* Allow excluding certain modules from lazy imports.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    15
* Expose an interface that's substantially the same as demandimport for
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    16
  Python 2.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    17
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    18
This also has some limitations compared to the Python 2 implementation:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    19
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    20
* Much of the logic is per-package, not per-module, so any packages loaded
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    21
  before demandimport is enabled will not be lazily imported in the future. In
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    22
  practice, we only expect builtins to be loaded before demandimport is
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    23
  enabled.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    24
"""
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    25
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    26
# This line is unnecessary, but it satisfies test-check-py3-compat.t.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    27
from __future__ import absolute_import
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    28
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    29
import contextlib
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    30
import importlib.abc
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    31
import importlib.machinery
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    32
import importlib.util
33898
3595e4e0ae57 demandimportpy3: update to pass import checker
Augie Fackler <raf@durin42.com>
parents: 33859
diff changeset
    33
import sys
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    34
42474
adb636392b3f demandimport: add tracing coverage for Python 3
Augie Fackler <augie@google.com>
parents: 37843
diff changeset
    35
from . import tracing
adb636392b3f demandimport: add tracing coverage for Python 3
Augie Fackler <augie@google.com>
parents: 37843
diff changeset
    36
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    37
_deactivated = False
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    38
44117
c5e0a9b97b8a hgdemandimport: disable on Python 3.5
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43378
diff changeset
    39
# Python 3.5's LazyLoader doesn't work for some reason.
c5e0a9b97b8a hgdemandimport: disable on Python 3.5
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43378
diff changeset
    40
# https://bugs.python.org/issue26186 is a known issue with extension
c5e0a9b97b8a hgdemandimport: disable on Python 3.5
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43378
diff changeset
    41
# importing. But it appears to not have a meaningful effect with
c5e0a9b97b8a hgdemandimport: disable on Python 3.5
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43378
diff changeset
    42
# Mercurial.
c5e0a9b97b8a hgdemandimport: disable on Python 3.5
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43378
diff changeset
    43
_supported = sys.version_info[0:2] >= (3, 6)
c5e0a9b97b8a hgdemandimport: disable on Python 3.5
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43378
diff changeset
    44
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    45
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    46
class _lazyloaderex(importlib.util.LazyLoader):
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    47
    """This is a LazyLoader except it also follows the _deactivated global and
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    48
    the ignore list.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    49
    """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    50
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    51
    def exec_module(self, module):
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    52
        """Make the module load lazily."""
42474
adb636392b3f demandimport: add tracing coverage for Python 3
Augie Fackler <augie@google.com>
parents: 37843
diff changeset
    53
        with tracing.log('demandimport %s', module):
adb636392b3f demandimport: add tracing coverage for Python 3
Augie Fackler <augie@google.com>
parents: 37843
diff changeset
    54
            if _deactivated or module.__name__ in ignores:
adb636392b3f demandimport: add tracing coverage for Python 3
Augie Fackler <augie@google.com>
parents: 37843
diff changeset
    55
                self.loader.exec_module(module)
adb636392b3f demandimport: add tracing coverage for Python 3
Augie Fackler <augie@google.com>
parents: 37843
diff changeset
    56
            else:
adb636392b3f demandimport: add tracing coverage for Python 3
Augie Fackler <augie@google.com>
parents: 37843
diff changeset
    57
                super().exec_module(module)
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    58
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    59
44117
c5e0a9b97b8a hgdemandimport: disable on Python 3.5
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43378
diff changeset
    60
_extensions_loader = _lazyloaderex.factory(
c5e0a9b97b8a hgdemandimport: disable on Python 3.5
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43378
diff changeset
    61
    importlib.machinery.ExtensionFileLoader
c5e0a9b97b8a hgdemandimport: disable on Python 3.5
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43378
diff changeset
    62
)
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    63
_bytecode_loader = _lazyloaderex.factory(
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    64
    importlib.machinery.SourcelessFileLoader
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    65
)
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    66
_source_loader = _lazyloaderex.factory(importlib.machinery.SourceFileLoader)
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    67
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    68
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    69
def _makefinder(path):
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    70
    return importlib.machinery.FileFinder(
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    71
        path,
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    72
        # This is the order in which loaders are passed in in core Python.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    73
        (_extensions_loader, importlib.machinery.EXTENSION_SUFFIXES),
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    74
        (_source_loader, importlib.machinery.SOURCE_SUFFIXES),
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    75
        (_bytecode_loader, importlib.machinery.BYTECODE_SUFFIXES),
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    76
    )
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    77
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    78
37843
670eb4fa1b86 demandimport: make module ignores a set (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35524
diff changeset
    79
ignores = set()
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    80
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    81
37843
670eb4fa1b86 demandimport: make module ignores a set (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35524
diff changeset
    82
def init(ignoreset):
670eb4fa1b86 demandimport: make module ignores a set (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35524
diff changeset
    83
    global ignores
670eb4fa1b86 demandimport: make module ignores a set (API)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35524
diff changeset
    84
    ignores = ignoreset
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    85
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    86
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    87
def isenabled():
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    88
    return _makefinder in sys.path_hooks and not _deactivated
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    89
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    90
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    91
def disable():
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    92
    try:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    93
        while True:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    94
            sys.path_hooks.remove(_makefinder)
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    95
    except ValueError:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    96
        pass
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    97
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
    98
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
    99
def enable():
44117
c5e0a9b97b8a hgdemandimport: disable on Python 3.5
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43378
diff changeset
   100
    if not _supported:
c5e0a9b97b8a hgdemandimport: disable on Python 3.5
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43378
diff changeset
   101
        return
c5e0a9b97b8a hgdemandimport: disable on Python 3.5
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43378
diff changeset
   102
33859
8fb5212652ec demandimport: move HGDEMANDIMPORT test to __init__.py
Jun Wu <quark@fb.com>
parents: 32423
diff changeset
   103
    sys.path_hooks.insert(0, _makefinder)
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   104
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42474
diff changeset
   105
32423
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   106
@contextlib.contextmanager
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   107
def deactivated():
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   108
    # This implementation is a bit different from Python 2's. Python 3
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   109
    # maintains a per-package finder cache in sys.path_importer_cache (see
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   110
    # PEP 302). This means that we can't just call disable + enable.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   111
    # If we do that, in situations like:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   112
    #
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   113
    #   demandimport.enable()
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   114
    #   ...
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   115
    #   from foo.bar import mod1
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   116
    #   with demandimport.deactivated():
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   117
    #       from foo.bar import mod2
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   118
    #
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   119
    # mod2 will be imported lazily. (The converse also holds -- whatever finder
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   120
    # first gets cached will be used.)
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   121
    #
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   122
    # Instead, have a global flag the LazyLoader can use.
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   123
    global _deactivated
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   124
    demandenabled = isenabled()
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   125
    if demandenabled:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   126
        _deactivated = True
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   127
    try:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   128
        yield
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   129
    finally:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   130
        if demandenabled:
859496bb6db3 demandimport: add python 3 implementation
Siddharth Agarwal <sid0@fb.com>
parents:
diff changeset
   131
            _deactivated = False