perf: clear vfs audit_cache before each run
When generating a stream clone, we spend a large amount of time auditing path.
Before this changes, the first run was warming the vfs cache for the other
runs, leading to a large runtime difference and a "faulty" reported timing for
the operation.
We now clear this important cache between run to get a more realistic timing.
Below are some example of median time change when clearing these cases. The
maximum time for a run did not changed significantly.
### data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog
# benchmark.name = hg.perf.exchange.stream.generate
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.version = latest
no-clearing: 17.289905
cache-clearing: 21.587965 (+24.86%, +4.30)
## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog
no-clearing: 32.670748
cache-clearing: 40.467095 (+23.86%, +7.80)
## data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
no-clearing: 37.838858
cache-clearing: 46.072749 (+21.76%, +8.23)
## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog
no-clearing: 32.969395
cache-clearing: 39.646209 (+20.25%, +6.68)
In addition, this significantly reduce the timing difference between the
performance command, from the perf extensions and a `real `hg bundle` call
producing a stream bundle. Some significant differences remain especially on
the "mozilla-try" repositories, but they are now smaller.
Note that some of that difference will actually not be
attributable to the stream generation (like maybe phases or branch map
computation).
Below are some benchmarks done on a currently draft changeset fixing some
unrelated slowness in `hg bundle` (34a78972af409d1ff37c29e60f6ca811ad1a457d)
### data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
hg.perf.exchange.stream.generate: 21.587965
hg.command.bundle: 24.301799 (+12.57%, +2.71)
## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog
hg.perf.exchange.stream.generate: 40.467095
hg.command.bundle: 44.831317 (+10.78%, +4.36)
## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog
hg.perf.exchange.stream.generate: 39.646209
hg.command.bundle: 45.395258 (+14.50%, +5.75)
## data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
hg.perf.exchange.stream.generate: 46.072749
hg.command.bundle: 55.882608 (+21.29%, +9.81)
## data-env-vars.name = mozilla-try-2023-03-22-zlib-general-delta
hg.perf.exchange.stream.generate: 334.716708
hg.command.bundle: 377.856767 (+12.89%, +43.14)
## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
hg.perf.exchange.stream.generate: 302.972301
hg.command.bundle: 326.098755 (+7.63%, +23.13)
== New Features ==
* clonebundles can be annotated with the expected memory requirements
using the `REQUIREDRAM` option. This allows clients to skip
bundles created with large zstd windows and fallback to larger, but
less demanding bundles.
* The `phabricator` extension now provides more functionality of the
arcanist CLI like changing the status of a differential.
* Phases processing is much faster, especially for repositories with
old non-public changesets.
== New Experimental Features ==
* The core of some hg operations have been (and are being)
implemented in rust, for speed. `hg status` on a repository with
300k tracked files goes from 1.8s to 0.6s for instance.
This has currently been tested only on linux, and does not build on
windows. See rust/README.rst in the mercurial repository for
instructions to opt into this.
* An experimental config `rewrite.empty-successor` was introduced to control
what happens when rewrite operations result in empty changesets.
== Bug Fixes ==
* For the case when connected to a TTY, stdout was fixed to be line-buffered
on Python 3 (where it was block-buffered before, causing the process to seem
hanging) and Windows on Python 2 (where it was unbuffered before).
* Subversion sources of the convert extension were fixed to work on Python 3.
* Subversion sources of the convert extension now interpret the encoding of
URLs like Subversion. Previously, there were situations where the convert
extension recognized a repository as present but Subversion did not, and
vice versa.
* The empty changeset check of in-memory rebases was fixed to match that of
normal rebases (and that of the commit command).
* The push command now checks the correct set of outgoing changesets for
obsolete and unstable changesets. Previously, it could happen that the check
prevented pushing changesets which were already on the server.
== Backwards Compatibility Changes ==
* Mercurial now requires at least Python 2.7.9 or a Python version that
backported modern SSL/TLS features (as defined in PEP 466), and that Python
was compiled against a OpenSSL version supporting TLS 1.1 or TLS 1.2
(likely this requires the OpenSSL version to be at least 1.0.1).
* The `hg perfwrite` command from contrib/perf.py was made more flexible and
changed its default behavior. To get the previous behavior, run `hg perfwrite
--nlines=100000 --nitems=1 --item='Testing write performance' --batch-line`.
* The absorb extension now preserves changesets with no file changes that can
be created by the commit command (those which change the branch name
compared to the parent and those closing a branch head).
== Internal API Changes ==
* logcmdutil.diffordiffstat() now takes contexts instead of nodes.
* The `mergestate` class along with some related methods and constants have
moved from `mercurial.merge` to a new `mercurial.mergestate` module.
* The `phasecache` class now uses sparse dictionaries for the phase data.
New accessors are provided to detect if any non-public changeset exists
(`hasnonpublicphases`) and get the correponsponding root set
(`nonpublicphaseroots`).
* The `stdin`, `stdout` and `stderr` attributes of the `mercurial.pycompat`
module were removed. Instead, the attributes of same name from the
`mercurial.utils.procutil` module should be used, which provide more
consistent behavior across Python versions and platforms.