hgdemandimport: apply lazy module loading to sys.meta_path finders Python's `sys.meta_path` finders are the primary objects whose job it is to find a module at import time. When `import` is called, Python iterates objects in this list and calls `o.find_spec(...)` to find a `ModuleSpec` (or None if the module couldn't be found by that finder). If no meta path finder can find a module, import fails. One of the default meta path finders is `PathFinder`. Its job is to import modules from the filesystem and is probably the most important importer. This finder looks at `sys.path` and `sys.path_hooks` to do its job. The `ModuleSpec` returned by `MetaPathImporter.find_spec()` has a `loader` attribute, which defines the concrete module loader to use. `sys.path_hooks` is a hook point for teaching `PathFinder` to instantiate custom loader types. Previously, we injected a custom `sys.path_hook` that told `PathFinder` to wrap the default loaders with a loader that creates a module object that is lazy. This approach worked. But its main limitation was that it only applied to the `PathFinder` meta path importer. There are other meta path importers that are registered. And in the case of PyOxidizer loading modules from memory, `PathFinder` doesn't come into play since PyOxidizer's own meta path importer was handling all imports. This commit changes our approach to lazy module loading by proxying all meta path importers. Specifically, we overload the `find_spec()` method to swap in a wrapped loader on the `ModuleSpec` before it is returned. The end result of this is all meta path importers should be lazy. As much as I would have loved to utilize .__class__ manipulation to achieve this, some meta path importers are implemented in C/Rust in such a way that they cannot be monkeypatched. This is why we use __getattribute__ to define a proxy. Also, this change could theoretically open us up to regressions in meta path importers whose loader is creating module objects which can't be monkeypatched. But I'm not aware of any of these in the wild. So I think we'll be safe. According to hyperfine, this change yields a decent startup time win of 5-6ms: ``` Benchmark #1: ~/.pyenv/versions/3.6.10/bin/python ./hg version Time (mean ± σ): 86.8 ms ± 0.5 ms [User: 78.0 ms, System: 8.7 ms] Range (min … max): 86.0 ms … 89.1 ms 50 runs Time (mean ± σ): 81.1 ms ± 2.7 ms [User: 74.5 ms, System: 6.5 ms] Range (min … max): 77.8 ms … 90.5 ms 50 runs Benchmark #2: ~/.pyenv/versions/3.7.6/bin/python ./hg version Time (mean ± σ): 78.9 ms ± 0.6 ms [User: 70.2 ms, System: 8.7 ms] Range (min … max): 78.1 ms … 81.2 ms 50 runs Time (mean ± σ): 73.4 ms ± 0.6 ms [User: 65.3 ms, System: 8.0 ms] Range (min … max): 72.4 ms … 75.7 ms 50 runs Benchmark #3: ~/.pyenv/versions/3.8.1/bin/python ./hg version Time (mean ± σ): 78.1 ms ± 0.6 ms [User: 70.2 ms, System: 7.9 ms] Range (min … max): 77.4 ms … 80.9 ms 50 runs Time (mean ± σ): 72.1 ms ± 0.4 ms [User: 64.4 ms, System: 7.6 ms] Range (min … max): 71.4 ms … 74.1 ms 50 runs ``` Differential Revision: https://phab.mercurial-scm.org/D7954

     1

# demandimportpy3 - global demand-loading of modules for Mercurial

     2

#

     3

# Copyright 2017 Facebook Inc.

     4

#

     5

# This software may be used and distributed according to the terms of the

     6

# GNU General Public License version 2 or any later version.

     7

     8

"""Lazy loading for Python 3.6 and above.

     9

    10

This uses the new importlib finder/loader functionality available in Python 3.5

    11

and up. The code reuses most of the mechanics implemented inside importlib.util,

    12

but with a few additions:

    13

    14

* Allow excluding certain modules from lazy imports.

    15

* Expose an interface that's substantially the same as demandimport for

    16

  Python 2.

    17

    18

This also has some limitations compared to the Python 2 implementation:

    19

    20

* Much of the logic is per-package, not per-module, so any packages loaded

    21

  before demandimport is enabled will not be lazily imported in the future. In

    22

  practice, we only expect builtins to be loaded before demandimport is

    23

  enabled.

    24

"""

    25

    26

# This line is unnecessary, but it satisfies test-check-py3-compat.t.

    27

from __future__ import absolute_import

    28

    29

import contextlib

    30

import importlib.util

    31

import sys

    32

    33

from . import tracing

    34

    35

_deactivated = False

    36

    37

# Python 3.5's LazyLoader doesn't work for some reason.

    38

# https://bugs.python.org/issue26186 is a known issue with extension

    39

# importing. But it appears to not have a meaningful effect with

    40

# Mercurial.

    41

_supported = sys.version_info[0:2] >= (3, 6)

    42

    43

    44

class _lazyloaderex(importlib.util.LazyLoader):

    45

    """This is a LazyLoader except it also follows the _deactivated global and

    46

    the ignore list.

    47

    """

    48

    49

    def exec_module(self, module):

    50

        """Make the module load lazily."""

    51

        with tracing.log('demandimport %s', module):

    52

            if _deactivated or module.__name__ in ignores:

    53

                self.loader.exec_module(module)

    54

            else:

    55

                super().exec_module(module)

    56

    57

    58

class LazyFinder(object):

    59

    """A wrapper around a ``MetaPathFinder`` that makes loaders lazy.

    60

    61

    ``sys.meta_path`` finders have their ``find_spec()`` called to locate a

    62

    module. This returns a ``ModuleSpec`` if found or ``None``. The

    63

    ``ModuleSpec`` has a ``loader`` attribute, which is called to actually

    64

    load a module.

    65

    66

    Our class wraps an existing finder and overloads its ``find_spec()`` to

    67

    replace the ``loader`` with our lazy loader proxy.

    68

    69

    We have to use __getattribute__ to proxy the instance because some meta

    70

    path finders don't support monkeypatching.

    71

    """

    72

    73

    __slots__ = ("_finder",)

    74

    75

    def __init__(self, finder):

    76

        object.__setattr__(self, "_finder", finder)

    77

    78

    def __repr__(self):

    79

        return "<LazyFinder for %r>" % object.__getattribute__(self, "_finder")

    80

    81

    # __bool__ is canonical Python 3. But check-code insists on __nonzero__ being

    82

    # defined via `def`.

    83

    def __nonzero__(self):

    84

        return bool(object.__getattribute__(self, "_finder"))

    85

    86

    __bool__ = __nonzero__

    87

    88

    def __getattribute__(self, name):

    89

        if name in ("_finder", "find_spec"):

    90

            return object.__getattribute__(self, name)

    91

    92

        return getattr(object.__getattribute__(self, "_finder"), name)

    93

    94

    def __delattr__(self, name):

    95

        return delattr(object.__getattribute__(self, "_finder"))

    96

    97

    def __setattr__(self, name, value):

    98

        return setattr(object.__getattribute__(self, "_finder"), name, value)

    99

   100

    def find_spec(self, *args, **kwargs):

   101

        finder = object.__getattribute__(self, "_finder")

   102

        spec = finder.find_spec(*args, **kwargs)

   103

   104

        # Lazy loader requires exec_module().

   105

        if (

   106

            spec is not None

   107

            and spec.loader is not None

   108

            and getattr(spec.loader, "exec_module")

   109

        ):

   110

            spec.loader = _lazyloaderex(spec.loader)

   111

   112

        return spec

   113

   114

   115

ignores = set()

   116

   117

   118

def init(ignoreset):

   119

    global ignores

   120

    ignores = ignoreset

   121

   122

   123

def isenabled():

   124

    return not _deactivated and any(

   125

        isinstance(finder, LazyFinder) for finder in sys.meta_path

   126

    )

   127

   128

   129

def disable():

   130

    new_finders = []

   131

    for finder in sys.meta_path:

   132

        new_finders.append(

   133

            finder._finder if isinstance(finder, LazyFinder) else finder

   134

        )

   135

    sys.meta_path[:] = new_finders

   136

   137

   138

def enable():

   139

    if not _supported:

   140

        return

   141

   142

    new_finders = []

   143

    for finder in sys.meta_path:

   144

        new_finders.append(

   145

            LazyFinder(finder) if not isinstance(finder, LazyFinder) else finder

   146

        )

   147

    sys.meta_path[:] = new_finders

   148

   149

   150

@contextlib.contextmanager

   151

def deactivated():

   152

    # This implementation is a bit different from Python 2's. Python 3

   153

    # maintains a per-package finder cache in sys.path_importer_cache (see

   154

    # PEP 302). This means that we can't just call disable + enable.

   155

    # If we do that, in situations like:

   156

    #

   157

    #   demandimport.enable()

   158

    #   ...

   159

    #   from foo.bar import mod1

   160

    #   with demandimport.deactivated():

   161

    #       from foo.bar import mod2

   162

    #

   163

    # mod2 will be imported lazily. (The converse also holds -- whatever finder

   164

    # first gets cached will be used.)

   165

    #

   166

    # Instead, have a global flag the LazyLoader can use.

   167

    global _deactivated

   168

    demandenabled = isenabled()

   169

    if demandenabled:

   170

        _deactivated = True

   171

    try:

   172

        yield

   173

    finally:

   174

        if demandenabled:

   175

            _deactivated = False