tests/filterpyflakes.py
author Pierre-Yves David <pierre-yves.david@octobus.net>
Fri, 05 Apr 2024 11:05:54 +0200
changeset 51576 de5bf3fe0233
parent 50433 e07dc1e7a454
permissions -rwxr-xr-x
revset: stop serializing node when using "%ln" Turning hundred of thousand of node from node to hex and back can be slow… what about we stop doing it? In many case were we are using node id we should be using revision id. However this is not a good reason to have a stupidly slow implementation of "%ln". This caught my attention again because the phase discovery during push make an extensive use of "%ln" or huge set. In absolute, that phase discovery probably should use "%ld" and need to improves its algorithmic complexity, but improving "%ln" seems simple and long overdue. This greatly speeds up `hg push` on repository with many drafts. Here are some relevant poulpe benchmarks: ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # benchmark.name = hg.command.push # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.explicit-rev = all-out-heads # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = default ## benchmark.variants.revs = any-1-extra-rev before: 44.235070 after: 20.416329 (-53.85%, -23.82) ## benchmark.variants.revs = any-100-extra-rev before: 49.234697 after: 26.519829 (-46.14%, -22.71) ### benchmark.name = hg.command.bundle # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.revs = all # benchmark.variants.type = none-streamv2 ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog before: 10.138396 after: 7.750458 (-23.55%, -2.39) ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog before: 1.263859 after: 0.700229 (-44.60%, -0.56) ## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog before: 399.484481 after: 346.5089 (-13.26%, -52.98) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog before: 4.540080 after: 3.401700 (-25.07%, -1.14) ## data-env-vars.name = tryton-public-2024-03-22-zstd-sparse-revlog before: 2.975765 after: 1.870798 (-37.13%, -1.10)
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
45830
c102b704edb5 global: use python3 in shebangs
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43076
diff changeset
     1
#!/usr/bin/env python3
14140
82f0412ef7de tests: add pyflakes checking for unused imports
timeless <timeless@mozdev.org>
parents:
diff changeset
     2
82f0412ef7de tests: add pyflakes checking for unused imports
timeless <timeless@mozdev.org>
parents:
diff changeset
     3
# Filter output by pyflakes to control which warnings we check
82f0412ef7de tests: add pyflakes checking for unused imports
timeless <timeless@mozdev.org>
parents:
diff changeset
     4
27285
aef5b606d3ee tests/filterpyflakes: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26023
diff changeset
     5
aef5b606d3ee tests/filterpyflakes: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26023
diff changeset
     6
import re
aef5b606d3ee tests/filterpyflakes: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26023
diff changeset
     7
import sys
14140
82f0412ef7de tests: add pyflakes checking for unused imports
timeless <timeless@mozdev.org>
parents:
diff changeset
     8
14173
419539ea79cb test-pyflake: improve sorting algorithm
timeless <timeless@mozdev.org>
parents: 14140
diff changeset
     9
lines = []
14140
82f0412ef7de tests: add pyflakes checking for unused imports
timeless <timeless@mozdev.org>
parents:
diff changeset
    10
for line in sys.stdin:
30421
21772a6a7861 filterpyflakes: dramatically simplify the entire thing by blacklisting
Augie Fackler <augie@google.com>
parents: 30394
diff changeset
    11
    # We blacklist tests that are too noisy for us
14175
b452abffcb15 tests: add pyflakes checking for assigned to but never used
timeless <timeless@mozdev.org>
parents: 14173
diff changeset
    12
    pats = [
32277
80e3002cd29e tests: remove special handling for undefined memoryview
Gregory Szorc <gregory.szorc@gmail.com>
parents: 30421
diff changeset
    13
        r"undefined name 'WindowsError'",
30421
21772a6a7861 filterpyflakes: dramatically simplify the entire thing by blacklisting
Augie Fackler <augie@google.com>
parents: 30394
diff changeset
    14
        r"redefinition of unused '[^']+' from line",
32510
50eaccb8353f filterpyflakes: allow reexporting pure symbols from cffi modules
Yuya Nishihara <yuya@tcha.org>
parents: 32277
diff changeset
    15
        # for cffi, allow re-exports from pure.*
50eaccb8353f filterpyflakes: allow reexporting pure symbols from cffi modules
Yuya Nishihara <yuya@tcha.org>
parents: 32277
diff changeset
    16
        r"cffi/[^:]*:.*\bimport \*' used",
50eaccb8353f filterpyflakes: allow reexporting pure symbols from cffi modules
Yuya Nishihara <yuya@tcha.org>
parents: 32277
diff changeset
    17
        r"cffi/[^:]*:.*\*' imported but unused",
30421
21772a6a7861 filterpyflakes: dramatically simplify the entire thing by blacklisting
Augie Fackler <augie@google.com>
parents: 30394
diff changeset
    18
    ]
21293
507ce509fd22 filterpyflakes: make memoryview filtering unconditional
Matt Mackall <mpm@selenic.com>
parents: 21271
diff changeset
    19
30421
21772a6a7861 filterpyflakes: dramatically simplify the entire thing by blacklisting
Augie Fackler <augie@google.com>
parents: 30394
diff changeset
    20
    keep = True
21772a6a7861 filterpyflakes: dramatically simplify the entire thing by blacklisting
Augie Fackler <augie@google.com>
parents: 30394
diff changeset
    21
    for pat in pats:
21772a6a7861 filterpyflakes: dramatically simplify the entire thing by blacklisting
Augie Fackler <augie@google.com>
parents: 30394
diff changeset
    22
        if re.search(pat, line):
21772a6a7861 filterpyflakes: dramatically simplify the entire thing by blacklisting
Augie Fackler <augie@google.com>
parents: 30394
diff changeset
    23
            keep = False
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 33367
diff changeset
    24
            break  # pattern matches
30421
21772a6a7861 filterpyflakes: dramatically simplify the entire thing by blacklisting
Augie Fackler <augie@google.com>
parents: 30394
diff changeset
    25
    if keep:
21772a6a7861 filterpyflakes: dramatically simplify the entire thing by blacklisting
Augie Fackler <augie@google.com>
parents: 30394
diff changeset
    26
        fn = line.split(':', 1)[0]
50433
e07dc1e7a454 tests: in filterpyflakes, tolerate non-ascii file contents
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 48875
diff changeset
    27
        with open(fn, 'rb') as f:
e07dc1e7a454 tests: in filterpyflakes, tolerate non-ascii file contents
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 48875
diff changeset
    28
            data = f.read()
e07dc1e7a454 tests: in filterpyflakes, tolerate non-ascii file contents
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 48875
diff changeset
    29
        if b'no-' b'check-code' in data:
30421
21772a6a7861 filterpyflakes: dramatically simplify the entire thing by blacklisting
Augie Fackler <augie@google.com>
parents: 30394
diff changeset
    30
            continue
21772a6a7861 filterpyflakes: dramatically simplify the entire thing by blacklisting
Augie Fackler <augie@google.com>
parents: 30394
diff changeset
    31
        lines.append(line)
14173
419539ea79cb test-pyflake: improve sorting algorithm
timeless <timeless@mozdev.org>
parents: 14140
diff changeset
    32
30421
21772a6a7861 filterpyflakes: dramatically simplify the entire thing by blacklisting
Augie Fackler <augie@google.com>
parents: 30394
diff changeset
    33
for line in lines:
14140
82f0412ef7de tests: add pyflakes checking for unused imports
timeless <timeless@mozdev.org>
parents:
diff changeset
    34
    sys.stdout.write(line)
28724
cf339d6ac7c7 py3: use print_function in filterpyflakes.py
Robert Stanca <robert.stanca7@gmail.com>
parents: 27285
diff changeset
    35
print()