tests/test-custom-filters.t
author Pierre-Yves David <pierre-yves.david@octobus.net>
Tue, 09 Apr 2024 02:54:19 +0200
changeset 51586 1cef1412af3e
parent 49621 55c6ebd11cb9
permissions -rw-r--r--
phases: rework the logic of _pushdiscoveryphase to bound complexity This rework the various graph traversal in _pushdiscoveryphase to keep the complexity in check. This is done though a couple of things: - first, limiting the space we have to explore, for example, if we are not in publishing push, we don't need to consider remote draft roots that are also draft locally, as there is nothing to be moved there. - avoid unbounded descendant computation, and use the faster "rev between" computation. This provide a massive boost to performance when exchanging with repository with a massive amount of draft, like mozilla-try: ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # benchmark.name = hg.command.push # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.explicit-rev = all-out-heads # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = default ## benchmark.variants.revs = any-1-extra-rev before: 20.346590 seconds after: 11.232059 seconds (-38.15%, -7.48 seconds) ## benchmark.variants.revs = any-100-extra-rev before: 24.752051 seconds after: 15.367412 seconds (-37.91%, -9.38 seconds) After this changes, the push operation is still quite too slow. Some of this can be attributed to general phases slowness (reading all the roots from disk for example) and other know slowness (not using persistent-nodemap, branchmap, tags, etc. We are also working on them, but with this series, phase discovery during push no longer showing up in profile and this is a pretty nice and bit low-hanging fruit out of the way. ### (same case as the above) # benchmark.variants.revs = any-1-extra-rev pre-%ln-change: 44.235070 this-changeset: 11.232059 seconds (-74.61%, -33.00 seconds) # benchmark.variants.revs = any-100-extra-rev pre-%ln-change: 49.234697 this-changeset: 15.367412 seconds (-68.79%, -33.87 seconds) Note that with this change, the `hg push` performance is now much closer to the `hg pull` performance, even it still lagging behind a bit. (and the overall performance are still too slow). ### data-env-vars.name = mozilla-try-2023-03-22-ds2-pnm # benchmark.variants.explicit-rev = all-out-heads # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.pulled-delta-reuse-policy = default # bin-env-vars.hg.flavor = rust ## benchmark.variants.revs = any-1-extra-rev hg.command.pull: 6.517450 hg.command.push: 11.219888 ## benchmark.variants.revs = any-100-extra-rev hg.command.pull: 10.160991 hg.command.push: 14.251107 ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # bin-env-vars.hg.py-re2-module = default # benchmark.variants.explicit-rev = all-out-heads # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.pulled-delta-reuse-policy = default ## bin-env-vars.hg.flavor = default ## benchmark.variants.revs = any-1-extra-rev hg.command.pull: 8.577772 hg.command.push: 11.232059 ## bin-env-vars.hg.flavor = default ## benchmark.variants.revs = any-100-extra-rev hg.command.pull: 13.152976 hg.command.push: 15.367412 ## bin-env-vars.hg.flavor = rust ## benchmark.variants.revs = any-1-extra-rev hg.command.pull: 8.731982 hg.command.push: 11.178751 ## bin-env-vars.hg.flavor = rust ## benchmark.variants.revs = any-100-extra-rev hg.command.pull: 13.184236 hg.command.push: 15.620843
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
49621
55c6ebd11cb9 tests: run many tests in $TESTTMP/repo instead of $TESTTMP
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 40609
diff changeset
     1
  $ hg init repo
55c6ebd11cb9 tests: run many tests in $TESTTMP/repo instead of $TESTTMP
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 40609
diff changeset
     2
  $ cd repo
6066
f3a8b5360100 Strip filter name from command before passing to filter function.
Jesse Glick <jesse.glick@sun.com>
parents:
diff changeset
     3
12124
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
     4
  $ cat > .hg/hgrc <<EOF
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
     5
  > [extensions]
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
     6
  > prefixfilter = prefix.py
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
     7
  > [encode]
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
     8
  > *.txt = stripprefix: Copyright 2046, The Masters
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
     9
  > [decode]
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    10
  > *.txt = insertprefix: Copyright 2046, The Masters
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    11
  > EOF
6066
f3a8b5360100 Strip filter name from command before passing to filter function.
Jesse Glick <jesse.glick@sun.com>
parents:
diff changeset
    12
12124
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    13
  $ cat > prefix.py <<EOF
26587
56b2bcea2529 error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 12316
diff changeset
    14
  > from mercurial import error
12124
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    15
  > def stripprefix(s, cmd, filename, **kwargs):
38071
6e713711331b py3: add b'' prefixes in tests/test-custom-filters.t
Pulkit Goyal <7895pulkit@gmail.com>
parents: 26587
diff changeset
    16
  >     header = b'%s\n' % cmd
12124
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    17
  >     if s[:len(header)] != header:
38071
6e713711331b py3: add b'' prefixes in tests/test-custom-filters.t
Pulkit Goyal <7895pulkit@gmail.com>
parents: 26587
diff changeset
    18
  >         raise error.Abort(b'missing header "%s" in %s' % (cmd, filename))
12124
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    19
  >     return s[len(header):]
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    20
  > def insertprefix(s, cmd):
38071
6e713711331b py3: add b'' prefixes in tests/test-custom-filters.t
Pulkit Goyal <7895pulkit@gmail.com>
parents: 26587
diff changeset
    21
  >     return b'%s\n%s' % (cmd, s)
12124
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    22
  > def reposetup(ui, repo):
38071
6e713711331b py3: add b'' prefixes in tests/test-custom-filters.t
Pulkit Goyal <7895pulkit@gmail.com>
parents: 26587
diff changeset
    23
  >     repo.adddatafilter(b'stripprefix:', stripprefix)
6e713711331b py3: add b'' prefixes in tests/test-custom-filters.t
Pulkit Goyal <7895pulkit@gmail.com>
parents: 26587
diff changeset
    24
  >     repo.adddatafilter(b'insertprefix:', insertprefix)
12124
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    25
  > EOF
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    26
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    27
  $ cat > .hgignore <<EOF
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    28
  > .hgignore
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    29
  > prefix.py
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    30
  > prefix.pyc
40609
ee9981bc8b44 py3: add __pycache__ to .hgignore
Pulkit Goyal <pulkit@yandex-team.ru>
parents: 38071
diff changeset
    31
  > __pycache__/
12124
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    32
  > EOF
6066
f3a8b5360100 Strip filter name from command before passing to filter function.
Jesse Glick <jesse.glick@sun.com>
parents:
diff changeset
    33
12124
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    34
  $ cat > stuff.txt <<EOF
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    35
  > Copyright 2046, The Masters
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    36
  > Some stuff to ponder very carefully.
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    37
  > EOF
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    38
  $ hg add stuff.txt
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    39
  $ hg ci -m stuff
6066
f3a8b5360100 Strip filter name from command before passing to filter function.
Jesse Glick <jesse.glick@sun.com>
parents:
diff changeset
    40
12124
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    41
Repository data:
6066
f3a8b5360100 Strip filter name from command before passing to filter function.
Jesse Glick <jesse.glick@sun.com>
parents:
diff changeset
    42
12124
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    43
  $ hg cat stuff.txt
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    44
  Some stuff to ponder very carefully.
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    45
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    46
Fresh checkout:
6066
f3a8b5360100 Strip filter name from command before passing to filter function.
Jesse Glick <jesse.glick@sun.com>
parents:
diff changeset
    47
12124
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    48
  $ rm stuff.txt
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    49
  $ hg up -C
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    50
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    51
  $ cat stuff.txt
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    52
  Copyright 2046, The Masters
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    53
  Some stuff to ponder very carefully.
12125
e5ccb9ea8266 test-custom-filter: heredoc is not required for single line operations
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 12124
diff changeset
    54
  $ echo "Very very carefully." >> stuff.txt
12124
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    55
  $ hg stat
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    56
  M stuff.txt
6066
f3a8b5360100 Strip filter name from command before passing to filter function.
Jesse Glick <jesse.glick@sun.com>
parents:
diff changeset
    57
12125
e5ccb9ea8266 test-custom-filter: heredoc is not required for single line operations
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 12124
diff changeset
    58
  $ echo "Unauthorized material subject to destruction." > morestuff.txt
12124
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    59
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    60
Problem encoding:
6066
f3a8b5360100 Strip filter name from command before passing to filter function.
Jesse Glick <jesse.glick@sun.com>
parents:
diff changeset
    61
12124
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    62
  $ hg add morestuff.txt
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    63
  $ hg ci -m morestuff
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    64
  abort: missing header "Copyright 2046, The Masters" in morestuff.txt
12316
4134686b83e1 tests: add exit codes to unified tests
Matt Mackall <mpm@selenic.com>
parents: 12125
diff changeset
    65
  [255]
12124
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    66
  $ hg stat
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    67
  M stuff.txt
7edda6c7985d tests: unify test-custom-filters
Nicolas Dumazet <nicdumz.commits@gmail.com>
parents: 8742
diff changeset
    68
  A morestuff.txt