contrib/import-checker.py
author Augie Fackler <raf@durin42.com>
Tue, 24 Dec 2013 19:10:04 -0500
changeset 20200 532fa12033e1
parent 20198 f5393a9dc4e5
child 20201 bc3b48b0f5c8
permissions -rw-r--r--
import-checker: use any() and a genexp to avoid awkward for/else construction
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     1
import ast
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     2
import os
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     3
import sys
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
     4
20198
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
     5
# Import a minimal set of stdlib modules needed for list_stdlib_modules()
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
     6
# to work when run from a virtualenv.  The modules were chosen empirically
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
     7
# so that the return value matches the return value without virtualenv.
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
     8
import BaseHTTPServer
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
     9
import zlib
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
    10
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    11
def dotted_name_of_path(path):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    12
    """Given a relative path to a source file, return its dotted module name.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    13
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    14
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    15
    >>> dotted_name_of_path('mercurial/error.py')
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    16
    'mercurial.error'
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    17
    """
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    18
    parts = path.split('/')
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    19
    parts[-1] = parts[-1][:-3] # remove .py
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    20
    return '.'.join(parts)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    21
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    22
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    23
def list_stdlib_modules():
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    24
    """List the modules present in the stdlib.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    25
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    26
    >>> mods = set(list_stdlib_modules())
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    27
    >>> 'BaseHTTPServer' in mods
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    28
    True
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    29
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    30
    os.path isn't really a module, so it's missing:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    31
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    32
    >>> 'os.path' in mods
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    33
    False
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    34
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    35
    sys requires special treatment, because it's baked into the
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    36
    interpreter, but it should still appear:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    37
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    38
    >>> 'sys' in mods
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    39
    True
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    40
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    41
    >>> 'collections' in mods
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    42
    True
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    43
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    44
    >>> 'cStringIO' in mods
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    45
    True
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    46
    """
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    47
    for m in sys.builtin_module_names:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    48
        yield m
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    49
    # These modules only exist on windows, but we should always
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    50
    # consider them stdlib.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    51
    for m in ['msvcrt', '_winreg']:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    52
        yield m
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    53
    # These get missed too
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    54
    for m in 'ctypes', 'email':
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    55
        yield m
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    56
    yield 'builtins' # python3 only
20197
761f2929a6ad import-checker: refactor sys.path prefix check (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20038
diff changeset
    57
    stdlib_prefixes = set([sys.prefix, sys.exec_prefix])
20198
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
    58
    # We need to supplement the list of prefixes for the search to work
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
    59
    # when run from within a virtualenv.
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
    60
    for mod in (BaseHTTPServer, zlib):
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
    61
        try:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
    62
            # Not all module objects have a __file__ attribute.
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
    63
            filename = mod.__file__
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
    64
        except AttributeError:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
    65
            continue
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
    66
        dirname = os.path.dirname(filename)
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
    67
        for prefix in stdlib_prefixes:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
    68
            if dirname.startswith(prefix):
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
    69
                # Then this directory is redundant.
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
    70
                break
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
    71
        else:
f5393a9dc4e5 import-checker: make test-module-imports.t work using virtualenv (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20197
diff changeset
    72
            stdlib_prefixes.add(dirname)
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    73
    for libpath in sys.path:
20197
761f2929a6ad import-checker: refactor sys.path prefix check (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20038
diff changeset
    74
        # We want to walk everything in sys.path that starts with something
761f2929a6ad import-checker: refactor sys.path prefix check (issue4129)
Chris Jerdonek <chris.jerdonek@gmail.com>
parents: 20038
diff changeset
    75
        # in stdlib_prefixes.
20200
532fa12033e1 import-checker: use any() and a genexp to avoid awkward for/else construction
Augie Fackler <raf@durin42.com>
parents: 20198
diff changeset
    76
        if not any(libpath.startswith(p) for p in stdlib_prefixes):
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    77
            continue
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    78
        if 'site-packages' in libpath:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    79
            continue
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    80
        for top, dirs, files in os.walk(libpath):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    81
            for name in files:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    82
                if name == '__init__.py':
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    83
                    continue
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    84
                if not (name.endswith('.py') or name.endswith('.so')):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    85
                    continue
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    86
                full_path = os.path.join(top, name)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    87
                if 'site-packages' in full_path:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    88
                    continue
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    89
                rel_path = full_path[len(libpath) + 1:]
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    90
                mod = dotted_name_of_path(rel_path)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    91
                yield mod
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    92
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    93
stdlib_modules = set(list_stdlib_modules())
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    94
20037
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
    95
def imported_modules(source, ignore_nested=False):
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    96
    """Given the source of a file as a string, yield the names
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    97
    imported by that file.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
    98
20037
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
    99
    Args:
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   100
      source: The python source to examine as a string.
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   101
      ignore_nested: If true, import statements that do not start in
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   102
                     column zero will be ignored.
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   103
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   104
    Returns:
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   105
      A list of module names imported by the given source.
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   106
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   107
    >>> sorted(imported_modules(
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   108
    ...         'import foo ; from baz import bar; import foo.qux'))
20037
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   109
    ['baz.bar', 'foo', 'foo.qux']
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   110
    >>> sorted(imported_modules(
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   111
    ... '''import foo
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   112
    ... def wat():
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   113
    ...     import bar
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   114
    ... ''', ignore_nested=True))
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   115
    ['foo']
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   116
    """
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   117
    for node in ast.walk(ast.parse(source)):
20037
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   118
        if ignore_nested and getattr(node, 'col_offset', 0) > 0:
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   119
            continue
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   120
        if isinstance(node, ast.Import):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   121
            for n in node.names:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   122
                yield n.name
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   123
        elif isinstance(node, ast.ImportFrom):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   124
            prefix = node.module + '.'
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   125
            for n in node.names:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   126
                yield prefix + n.name
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   127
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   128
def verify_stdlib_on_own_line(source):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   129
    """Given some python source, verify that stdlib imports are done
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   130
    in separate statements from relative local module imports.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   131
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   132
    Observing this limitation is important as it works around an
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   133
    annoying lib2to3 bug in relative import rewrites:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   134
    http://bugs.python.org/issue19510.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   135
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   136
    >>> list(verify_stdlib_on_own_line('import sys, foo'))
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   137
    ['mixed stdlib and relative imports:\\n   foo, sys']
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   138
    >>> list(verify_stdlib_on_own_line('import sys, os'))
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   139
    []
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   140
    >>> list(verify_stdlib_on_own_line('import foo, bar'))
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   141
    []
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   142
    """
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   143
    for node in ast.walk(ast.parse(source)):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   144
        if isinstance(node, ast.Import):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   145
            from_stdlib = {}
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   146
            for n in node.names:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   147
                from_stdlib[n.name] = n.name in stdlib_modules
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   148
            num_std = len([x for x in from_stdlib.values() if x])
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   149
            if num_std not in (len(from_stdlib.values()), 0):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   150
                yield ('mixed stdlib and relative imports:\n   %s' %
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   151
                       ', '.join(sorted(from_stdlib.iterkeys())))
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   152
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   153
class CircularImport(Exception):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   154
    pass
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   155
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   156
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   157
def cyclekey(names):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   158
    return tuple(sorted(set(names)))
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   159
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   160
def check_one_mod(mod, imports, path=None, ignore=None):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   161
    if path is None:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   162
        path = []
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   163
    if ignore is None:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   164
        ignore = []
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   165
    path = path + [mod]
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   166
    for i in sorted(imports.get(mod, [])):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   167
        if i not in stdlib_modules:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   168
            i = mod.rsplit('.', 1)[0] + '.' + i
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   169
        if i in path:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   170
            firstspot = path.index(i)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   171
            cycle = path[firstspot:] + [i]
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   172
            if cyclekey(cycle) not in ignore:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   173
                raise CircularImport(cycle)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   174
            continue
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   175
        check_one_mod(i, imports, path=path, ignore=ignore)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   176
20038
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
   177
def rotatecycle(cycle):
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
   178
    """arrange a cycle so that the lexicographically first module listed first
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
   179
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
   180
    >>> rotatecycle(['foo', 'bar', 'foo'])
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
   181
    ['bar', 'foo', 'bar']
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
   182
    """
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
   183
    lowest = min(cycle)
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
   184
    idx = cycle.index(lowest)
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
   185
    return cycle[idx:] + cycle[1:idx] + [lowest]
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   186
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   187
def find_cycles(imports):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   188
    """Find cycles in an already-loaded import graph.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   189
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   190
    >>> imports = {'top.foo': ['bar', 'os.path', 'qux'],
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   191
    ...            'top.bar': ['baz', 'sys'],
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   192
    ...            'top.baz': ['foo'],
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   193
    ...            'top.qux': ['foo']}
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   194
    >>> print '\\n'.join(sorted(find_cycles(imports)))
20038
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
   195
    top.bar -> top.baz -> top.foo -> top.bar -> top.bar
c65a6937b828 import-checker: try a little harder to show fewer cycles
Augie Fackler <raf@durin42.com>
parents: 20037
diff changeset
   196
    top.foo -> top.qux -> top.foo -> top.foo
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   197
    """
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   198
    cycles = {}
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   199
    for mod in sorted(imports.iterkeys()):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   200
        try:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   201
            check_one_mod(mod, imports, ignore=cycles)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   202
        except CircularImport, e:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   203
            cycle = e.args[0]
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   204
            cycles[cyclekey(cycle)] = ' -> '.join(rotatecycle(cycle))
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   205
    return cycles.values()
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   206
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   207
def _cycle_sortkey(c):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   208
    return len(c), c
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   209
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   210
def main(argv):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   211
    if len(argv) < 2:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   212
        print 'Usage: %s file [file] [file] ...'
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   213
        return 1
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   214
    used_imports = {}
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   215
    any_errors = False
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   216
    for source_path in argv[1:]:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   217
        f = open(source_path)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   218
        modname = dotted_name_of_path(source_path)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   219
        src = f.read()
20037
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   220
        used_imports[modname] = sorted(
957b43371928 import-checker: ignore nested imports
Augie Fackler <raf@durin42.com>
parents: 20036
diff changeset
   221
            imported_modules(src, ignore_nested=True))
20036
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   222
        for error in verify_stdlib_on_own_line(src):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   223
            any_errors = True
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   224
            print source_path, error
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   225
        f.close()
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   226
    cycles = find_cycles(used_imports)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   227
    if cycles:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   228
        firstmods = set()
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   229
        for c in sorted(cycles, key=_cycle_sortkey):
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   230
            first = c.split()[0]
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   231
            # As a rough cut, ignore any cycle that starts with the
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   232
            # same module as some other cycle. Otherwise we see lots
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   233
            # of cycles that are effectively duplicates.
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   234
            if first in firstmods:
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   235
                continue
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   236
            print 'Import cycle:', c
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   237
            firstmods.add(first)
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   238
        any_errors = True
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   239
    return not any_errors
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   240
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   241
if __name__ == '__main__':
e5d51fa51aba contrib: add an import checker
Augie Fackler <raf@durin42.com>
parents:
diff changeset
   242
    sys.exit(int(main(sys.argv)))