perf: clear vfs audit_cache before each run
When generating a stream clone, we spend a large amount of time auditing path.
Before this changes, the first run was warming the vfs cache for the other
runs, leading to a large runtime difference and a "faulty" reported timing for
the operation.
We now clear this important cache between run to get a more realistic timing.
Below are some example of median time change when clearing these cases. The
maximum time for a run did not changed significantly.
### data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog
# benchmark.name = hg.perf.exchange.stream.generate
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.version = latest
no-clearing: 17.289905
cache-clearing: 21.587965 (+24.86%, +4.30)
## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog
no-clearing: 32.670748
cache-clearing: 40.467095 (+23.86%, +7.80)
## data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
no-clearing: 37.838858
cache-clearing: 46.072749 (+21.76%, +8.23)
## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog
no-clearing: 32.969395
cache-clearing: 39.646209 (+20.25%, +6.68)
In addition, this significantly reduce the timing difference between the
performance command, from the perf extensions and a `real `hg bundle` call
producing a stream bundle. Some significant differences remain especially on
the "mozilla-try" repositories, but they are now smaller.
Note that some of that difference will actually not be
attributable to the stream generation (like maybe phases or branch map
computation).
Below are some benchmarks done on a currently draft changeset fixing some
unrelated slowness in `hg bundle` (34a78972af409d1ff37c29e60f6ca811ad1a457d)
### data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
hg.perf.exchange.stream.generate: 21.587965
hg.command.bundle: 24.301799 (+12.57%, +2.71)
## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog
hg.perf.exchange.stream.generate: 40.467095
hg.command.bundle: 44.831317 (+10.78%, +4.36)
## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog
hg.perf.exchange.stream.generate: 39.646209
hg.command.bundle: 45.395258 (+14.50%, +5.75)
## data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
hg.perf.exchange.stream.generate: 46.072749
hg.command.bundle: 55.882608 (+21.29%, +9.81)
## data-env-vars.name = mozilla-try-2023-03-22-zlib-general-delta
hg.perf.exchange.stream.generate: 334.716708
hg.command.bundle: 377.856767 (+12.89%, +43.14)
## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
hg.perf.exchange.stream.generate: 302.972301
hg.command.bundle: 326.098755 (+7.63%, +23.13)
========================================================
Test the ability to access a hidden revision on a server
========================================================
#require serve
$ . $TESTDIR/testlib/obsmarker-common.sh
$ cat >> $HGRCPATH << EOF
> [ui]
> ssh = "$PYTHON" "$RUNTESTDIR/dummyssh"
> [phases]
> # public changeset are not obsolete
> publish=false
> [experimental]
> evolution=all
> [ui]
> logtemplate='{rev}:{node|short} {desc} [{phase}]\n'
> EOF
Setup a simple repository with some hidden revisions
----------------------------------------------------
Testing the `served.hidden` view
$ hg init repo-with-hidden
$ cd repo-with-hidden
$ echo 0 > a
$ hg ci -qAm "c_Public"
$ hg phase --public
$ echo 1 > a
$ hg ci -m "c_Amend_Old"
$ echo 2 > a
$ hg ci -m "c_Amend_New" --amend
$ hg up ".^"
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ echo 3 > a
$ hg ci -m "c_Pruned"
created new head
$ hg debugobsolete --record-parents `getid 'desc("c_Pruned")'` -d '0 0'
1 new obsolescence markers
obsoleted 1 changesets
$ hg up ".^"
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ echo 4 > a
$ hg ci -m "c_Secret" --secret
created new head
$ echo 5 > a
$ hg ci -m "c_Secret_Pruned" --secret
$ hg debugobsolete --record-parents `getid 'desc("c_Secret_Pruned")'` -d '0 0'
1 new obsolescence markers
obsoleted 1 changesets
$ hg up null
0 files updated, 0 files merged, 1 files removed, 0 files unresolved
$ hg log -G -T '{rev}:{node|short} {desc} [{phase}]\n' --hidden
x 5:8d28cbe335f3 c_Secret_Pruned [secret]
|
o 4:1c6afd79eb66 c_Secret [secret]
|
| x 3:5d1575e42c25 c_Pruned [draft]
|/
| o 2:c33affeb3f6b c_Amend_New [draft]
|/
| x 1:be215fbb8c50 c_Amend_Old [draft]
|/
o 0:5f354f46e585 c_Public [public]
$ hg debugobsolete
be215fbb8c5090028b00154c1fe877ad1b376c61 c33affeb3f6b4e9621d1839d6175ddc07708807c 0 (Thu Jan 01 00:00:00 1970 +0000) {'ef1': '9', 'operation': 'amend', 'user': 'test'}
5d1575e42c25b7f2db75cd4e0b881b1c35158fae 0 {5f354f46e5853535841ec7a128423e991ca4d59b} (Thu Jan 01 00:00:00 1970 +0000) {'user': 'test'}
8d28cbe335f311bc89332d7bbe8a07889b6914a0 0 {1c6afd79eb6663275bbe30097e162b1c24ced0f0} (Thu Jan 01 00:00:00 1970 +0000) {'user': 'test'}
$ cd ..
Test the feature
================
Check cache pre-warm
--------------------
$ ls -1 repo-with-hidden/.hg/cache
branch2
branch2-base
branch2-served
branch2-served.hidden
branch2-visible
rbc-names-v1
rbc-revs-v1
tags2
tags2-visible
Check that the `served.hidden` repoview
---------------------------------------
$ hg -R repo-with-hidden serve -p $HGPORT -d --pid-file hg.pid --config web.view=served.hidden
$ cat hg.pid >> $DAEMON_PIDS
changesets in secret and higher phases are not visible through hgweb
$ hg -R repo-with-hidden log --template "revision: {rev}\\n" --rev "reverse(not secret())"
revision: 2
revision: 0
$ hg -R repo-with-hidden log --template "revision: {rev}\\n" --rev "reverse(not secret())" --hidden
revision: 3
revision: 2
revision: 1
revision: 0
$ get-with-headers.py localhost:$HGPORT 'log?style=raw' | grep revision:
revision: 3
revision: 2
revision: 1
revision: 0
$ killdaemons.py
Test --remote-hidden for local peer
-----------------------------------
$ hg clone --pull repo-with-hidden client
requesting all changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 2 changes to 1 files
2 new obsolescence markers
new changesets 5f354f46e585:c33affeb3f6b (1 drafts)
updating to branch default
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg -R client log -G --hidden -v
@ 1:c33affeb3f6b c_Amend_New [draft]
|
o 0:5f354f46e585 c_Public [public]
pulling an hidden changeset should fail:
$ hg -R client pull -r be215fbb8c50
pulling from $TESTTMP/repo-with-hidden
abort: filtered revision 'be215fbb8c50' (not in 'served' subset)
[10]
pulling an hidden changeset with --remote-hidden should succeed:
$ hg -R client pull --remote-hidden --traceback -r be215fbb8c50
pulling from $TESTTMP/repo-with-hidden
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files (+1 heads)
(1 other changesets obsolete on arrival)
(run 'hg heads' to see heads)
$ hg -R client log -G --hidden -v
x 2:be215fbb8c50 c_Amend_Old [draft]
|
| @ 1:c33affeb3f6b c_Amend_New [draft]
|/
o 0:5f354f46e585 c_Public [public]
Pulling a secret changeset is still forbidden:
secret visible:
$ hg -R client pull --remote-hidden -r 8d28cbe335f3
pulling from $TESTTMP/repo-with-hidden
abort: filtered revision '8d28cbe335f3' (not in 'served.hidden' subset)
[10]
secret hidden:
$ hg -R client pull --remote-hidden -r 1c6afd79eb66
pulling from $TESTTMP/repo-with-hidden
abort: filtered revision '1c6afd79eb66' (not in 'served.hidden' subset)
[10]
Test accessing hidden changeset through hgweb
---------------------------------------------
$ hg -R repo-with-hidden serve -p $HGPORT -d --pid-file hg.pid --config "experimental.server.allow-hidden-access=*" -E error.log --accesslog access.log
$ cat hg.pid >> $DAEMON_PIDS
Hidden changeset are hidden by default:
$ get-with-headers.py localhost:$HGPORT 'log?style=raw' | grep revision:
revision: 2
revision: 0
Hidden changeset are visible when requested:
$ get-with-headers.py localhost:$HGPORT 'log?style=raw&access-hidden=1' | grep revision:
revision: 3
revision: 2
revision: 1
revision: 0
Same check on a server that do not allow hidden access:
```````````````````````````````````````````````````````
$ hg -R repo-with-hidden serve -p $HGPORT1 -d --pid-file hg2.pid --config "experimental.server.allow-hidden-access=" -E error.log --accesslog access.log
$ cat hg2.pid >> $DAEMON_PIDS
Hidden changeset are hidden by default:
$ get-with-headers.py localhost:$HGPORT1 'log?style=raw' | grep revision:
revision: 2
revision: 0
Hidden changeset are still hidden despite being the hidden access request:
$ get-with-headers.py localhost:$HGPORT1 'log?style=raw&access-hidden=1' | grep revision:
revision: 2
revision: 0
Test --remote-hidden for http peer
----------------------------------
$ hg clone --pull http://localhost:$HGPORT client-http
requesting all changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 2 changes to 1 files
2 new obsolescence markers
new changesets 5f354f46e585:c33affeb3f6b (1 drafts)
updating to branch default
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg -R client-http log -G --hidden -v
@ 1:c33affeb3f6b c_Amend_New [draft]
|
o 0:5f354f46e585 c_Public [public]
pulling an hidden changeset should fail:
$ hg -R client-http pull -r be215fbb8c50
pulling from http://localhost:$HGPORT/
abort: filtered revision 'be215fbb8c50' (not in 'served' subset)
[255]
pulling an hidden changeset with --remote-hidden should succeed:
$ hg -R client-http pull --remote-hidden -r be215fbb8c50
pulling from http://localhost:$HGPORT/
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files (+1 heads)
(1 other changesets obsolete on arrival)
(run 'hg heads' to see heads)
$ hg -R client-http log -G --hidden -v
x 2:be215fbb8c50 c_Amend_Old [draft]
|
| @ 1:c33affeb3f6b c_Amend_New [draft]
|/
o 0:5f354f46e585 c_Public [public]
Pulling a secret changeset is still forbidden:
secret visible:
$ hg -R client-http pull --remote-hidden -r 8d28cbe335f3
pulling from http://localhost:$HGPORT/
abort: filtered revision '8d28cbe335f3' (not in 'served.hidden' subset)
[255]
secret hidden:
$ hg -R client-http pull --remote-hidden -r 1c6afd79eb66
pulling from http://localhost:$HGPORT/
abort: filtered revision '1c6afd79eb66' (not in 'served.hidden' subset)
[255]
Same check on a server that do not allow hidden access:
```````````````````````````````````````````````````````
$ hg clone --pull http://localhost:$HGPORT1 client-http2
requesting all changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 2 changes to 1 files
2 new obsolescence markers
new changesets 5f354f46e585:c33affeb3f6b (1 drafts)
updating to branch default
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg -R client-http2 log -G --hidden -v
@ 1:c33affeb3f6b c_Amend_New [draft]
|
o 0:5f354f46e585 c_Public [public]
pulling an hidden changeset should fail:
$ hg -R client-http2 pull -r be215fbb8c50
pulling from http://localhost:$HGPORT1/
abort: filtered revision 'be215fbb8c50' (not in 'served' subset)
[255]
pulling an hidden changeset with --remote-hidden should fail too:
$ hg -R client-http2 pull --remote-hidden -r be215fbb8c50
pulling from http://localhost:$HGPORT1/
abort: filtered revision 'be215fbb8c50' (not in 'served' subset)
[255]
Test --remote-hidden for ssh peer
----------------------------------
$ hg clone --pull ssh://user@dummy/repo-with-hidden client-ssh
requesting all changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 2 changes to 1 files
2 new obsolescence markers
new changesets 5f354f46e585:c33affeb3f6b (1 drafts)
updating to branch default
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg -R client-ssh log -G --hidden -v
@ 1:c33affeb3f6b c_Amend_New [draft]
|
o 0:5f354f46e585 c_Public [public]
Check on a server that do not allow hidden access:
``````````````````````````````````````````````````
pulling an hidden changeset should fail:
$ hg -R client-ssh pull -r be215fbb8c50
pulling from ssh://user@dummy/repo-with-hidden
abort: filtered revision 'be215fbb8c50' (not in 'served' subset)
[255]
pulling an hidden changeset with --remote-hidden should succeed:
$ hg -R client-ssh pull --remote-hidden -r be215fbb8c50
pulling from ssh://user@dummy/repo-with-hidden
remote: ignoring request to access hidden changeset by unauthorized user: * (glob)
abort: filtered revision 'be215fbb8c50' (not in 'served' subset)
[255]
$ hg -R client-ssh log -G --hidden -v
@ 1:c33affeb3f6b c_Amend_New [draft]
|
o 0:5f354f46e585 c_Public [public]
Check on a server that do allow hidden access:
``````````````````````````````````````````````
$ cat << EOF >> repo-with-hidden/.hg/hgrc
> [experimental]
> server.allow-hidden-access=*
> EOF
pulling an hidden changeset should fail:
$ hg -R client-ssh pull -r be215fbb8c50
pulling from ssh://user@dummy/repo-with-hidden
abort: filtered revision 'be215fbb8c50' (not in 'served' subset)
[255]
pulling an hidden changeset with --remote-hidden should succeed:
$ hg -R client-ssh pull --remote-hidden -r be215fbb8c50
pulling from ssh://user@dummy/repo-with-hidden
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files (+1 heads)
(1 other changesets obsolete on arrival)
(run 'hg heads' to see heads)
$ hg -R client-ssh log -G --hidden -v
x 2:be215fbb8c50 c_Amend_Old [draft]
|
| @ 1:c33affeb3f6b c_Amend_New [draft]
|/
o 0:5f354f46e585 c_Public [public]
Pulling a secret changeset is still forbidden:
secret visible:
$ hg -R client-ssh pull --remote-hidden -r 8d28cbe335f3
pulling from ssh://user@dummy/repo-with-hidden
abort: filtered revision '8d28cbe335f3' (not in 'served.hidden' subset)
[255]
secret hidden:
$ hg -R client-ssh pull --remote-hidden -r 1c6afd79eb66
pulling from ssh://user@dummy/repo-with-hidden
abort: filtered revision '1c6afd79eb66' (not in 'served.hidden' subset)
[255]
=============
Final cleanup
=============
$ killdaemons.py