Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 26 Apr 2024 19:10:35 +0100] rev 51626
dirstate: remove the python-side whitelist of allowed matchers
This whitelist is too permissive because it allows matchers that contain
disallowed ones deep inside, for example through `intersectionmatcher`.
It is also too restrictive because it doesn't pass through
some of the matchers we support, such as `patternmatcher`.
It's also unnecessary because unsupported matchers raise
`FallbackError` and we fall back anyway.
Making this change makes more of the tests use rust code path,
and therefore subtly change behavior. For example, rust status
in largefiles repos seems to have strange behavior.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 26 Apr 2024 18:53:02 +0100] rev 51625
match: make `was_tampered_with` work recursively
This is useful if we are to use it outside of Rust, when
deciding whether or not to do some fast-path operation with
a given matcher.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 26 Apr 2024 19:43:42 +0100] rev 51624
largefiles: mark more matchers as having been tampered with
These happened to slip through the cracks earlier because they
weren't caught by tests. Now that we're enabling rust fast path
more widely these start breaking.
Raphaël Gomès <rgomes@octobus.net> [Mon, 06 May 2024 18:50:21 +0200] rev 51623
branching: merge stable into default
Raphaël Gomès <rgomes@octobus.net> [Mon, 06 May 2024 18:48:37 +0200] rev 51622
Added signature for changeset 028dc3f92dbd
Raphaël Gomès <rgomes@octobus.net> [Mon, 06 May 2024 18:48:34 +0200] rev 51621
Added tag 6.7.3 for changeset 028dc3f92dbd
Raphaël Gomès <rgomes@octobus.net> [Mon, 06 May 2024 17:51:21 +0200] rev 51620
relnotes: add 6.7.3
Georges Racinet <georges.racinet@octobus.net> [Mon, 22 Apr 2024 19:47:08 +0200] rev 51619
rust: blanket implementation of Graph for Graph references
The need comes from the fact that `AncestorsIterator` and many
Graph-related algorithms take ownership of the `Graph` they work with.
This, in turn is due to them needing to accept the `Index` instances
that are provided by the Python layers (that neither rhg nor `RHGitaly`
use, of course): the fact that nowadays the Python layer holds an object
that is itself implemented in Rust does not change the core problem that
they cannot be tracked by the borrow checker.
Even though it looks like cloning `Changelog` would be cheap, it seems
hard to guarantee that on the long run. The object is already too rich
for us to be comfortable with it, when using references is the most
natural and guaranteed way of proceeding.
The added test seems a bit superfleous, but it will act as a reminder
that this feature is really useful until something in the Mercurial code
base actually uses it.
Raphaël Gomès <rgomes@octobus.net> [Mon, 06 May 2024 15:30:21 +0200] rev 51618
rust-cpython: don't swallow the dirstate error message
In case we do get a dirstate error, we want to get the full error message and
not just an opaque `Dirstate error`.
Raphaël Gomès <rgomes@octobus.net> [Mon, 06 May 2024 13:07:02 +0200] rev 51617
dirstate-v2: check that root nodes are at the root before writing
More explanations in the previous changeset.
Raphaël Gomès <rgomes@octobus.net> [Mon, 06 May 2024 13:02:52 +0200] rev 51616
dirstate-v2: add check of parent/child nodes being related when writing
This stems from a corruption seen in a private repository. We're not sure
of the source of the corruption, and it's very possible that we're seeing
compounded effects of multiple writes on a corrupted dirstate.
Adding this check is not expensive in itself and large writes of the dirstate
are not common.
This change does not catch this problem at the root node, the next one will.
Raphaël Gomès <rgomes@octobus.net> [Mon, 06 May 2024 16:29:00 +0200] rev 51615
admin-verify: expect a number of errors to be returned
It's the responsibility of the check to handle errors, we only care about
the total count to sum up the check's work.
We use `admin::verify -c dirstate` to test this path at least somewhat.
Raphaël Gomès <rgomes@octobus.net> [Mon, 06 May 2024 16:16:15 +0200] rev 51614
admin-verify: fix error message handling
`dirstate.verify` used to return tuples but does not anymore, it returns
the pre-formatted error message, which is a nicer interface anyway.
Raphaël Gomès <rgomes@octobus.net> [Mon, 06 May 2024 12:31:29 +0200] rev 51613
admin-verify: pass p1 down to the dirstate function
This was forgotten and can break with certain kinds of corruption.
Raphaël Gomès <rgomes@octobus.net> [Mon, 06 May 2024 11:27:29 +0200] rev 51612
Backed out changeset 3e0f86f09f26
Raphaël Gomès <rgomes@octobus.net> [Mon, 06 May 2024 11:26:52 +0200] rev 51611
Backed out changeset fc317bd5b637
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 May 2024 02:20:42 +0200] rev 51610
re2: make errors quiet
By default, the re2 library will output error on its own instead of keeping the
error in an exception. This make re2 printing spurious error before fallback to
the stdlib remodule that may accept the pattern or also fails to parse it and
raise a proper error that will be handled by Mercurial.
So we also pass an Option object that changes this default.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 May 2024 08:46:58 +0200] rev 51609
fold-or-prune-me: update proposal
This does the same things but with a narrower wrapping.
Felipe Resende <felipe@fcresende.dev.br> [Sun, 31 Mar 2024 17:57:46 -0300] rev 51608
subrepo: propagate non-default path on outgoing
There was already a fix made in 5dbff89cf107 for pull and push commands. I did
the same for the outgoing command.
The problem I identified is that when the parent repository has multiple paths,
the outgoing command was not respecting the parent path used and was always
using the default path for subrepositories.
Hraban Luyat <hraban@0brg.net> [Tue, 26 Mar 2024 01:27:27 -0400] rev 51607
hgrc: search XDG_CONFIG_HOME on mac
Searching for hgrc was special cased not to look through ~/.config/hg on Mac,
but that’s unnecessary: Macs support it as do other unix based systems. There
are plenty tools that use it there, e.g. git, and people expect it to work, e.g.
"https://stackoverflow.com/questions/72499837/mercurial-on-macos-doesnt-read-config-hg-hgrc".
Initial code introduced in 354020079723.
Raphaël Gomès <rgomes@octobus.net> [Tue, 16 Apr 2024 09:51:11 +0200] rev 51606
base-revsets: use an author that actually exercises a lot of changesets
This was caught in my big find-and-replace: d4ba4d51f85f.
The point of `base-revsets` is to give revsets that will give a good coverage
of the repository. Using Pierre-Yves as the second largest committer
(in terms of number of changesets) seems like a good idea.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Tue, 16 Apr 2024 17:21:37 +0100] rev 51605
match: simplify the rust-side file pattern kind parsing
There's no need to add the ':' characters if
we're simply pattern matching against constants next.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Tue, 16 Apr 2024 13:51:45 +0100] rev 51604
match: share code between includematcher and patternmatcher
No need to have this duplication.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 05 Apr 2024 17:57:26 +0100] rev 51603
matchers: support patternmatcher in rust
Arseniy Alekseyev <aalekseyev@janestreet.com> [Tue, 09 Apr 2024 11:12:24 +0100] rev 51602
match: avoid rust fast path if the matcher was tampered with
Otherwise the fast path does not respect the modifications made
by the extension (concretely largefiles, but other extensions can
start using that too)
Arseniy Alekseyev <aalekseyev@janestreet.com> [Tue, 09 Apr 2024 11:00:52 +0100] rev 51601
largefiles: track if a matcher was tampered with
This is used to make sure rust fast path is not taken for the
modified matchers.
Raphaël Gomès <rgomes@octobus.net> [Wed, 17 Apr 2024 12:28:48 +0200] rev 51600
branching: merge stable into default
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 13 Mar 2024 12:02:06 +0100] rev 51599
tags-cache: directly perform a monimal walk for hgtagsfnodescache warming
We do something narrower than the path retrieving data. So lets use dedicated
code instead.
This provides further useful speedup:
### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
# benchmark.name = hg.debug.debug-update-cache
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.pre-state = warm
before-this-series: 19.947581
skip-fnode-filter: 18.916804 (-5.17%, -1.03)
use-rev-num: 17.493725 (-12.30%, -2.45)
this-changesets: 15.919466 (-20.19%, -4.03)
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 13 Mar 2024 11:51:11 +0100] rev 51598
tags-cache: directly operate on rev-num warming hgtagsfnodescache
Not having to goes through nodeid speed up things notably.
### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
# benchmark.name = hg.debug.debug-update-cache
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.pre-state = warm
before-this-series: 19.947581
before-this-changes: 18.916804 (-5.17%, -1.03)
this-changesets: 17.493725 (-12.30%, -2.45)
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 13 Mar 2024 11:38:28 +0100] rev 51597
tags-cache: skip the filternode step if we are not going to use it
When warming the hgtagsfnodescache, we don't need the actual result, so we can
simply skip the part that "filter" fnode we read from the cache. So provide a
quite visible speed up to the top level `hg debugupdatecache` function.
### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
# benchmark.name = hg.debug.debug-update-cache
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.pre-state = warm
before: 19.947581
after: 18.916804 (-5.17%, -1.03)
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 13 Mar 2024 11:34:21 +0100] rev 51596
tags-cache: add a dedicated warm cache function to hgtagsfnodescache
Having a dedicated API point will help to optimize that specific usage. Right
doing a full phases weam takes a long time, even when the cache is already
filled.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 09 Apr 2024 22:37:15 +0200] rev 51595
outgoing: add a simple fastpath when there is no common
This further speed up case like `hg bundle --all` for larger repository.
### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
# benchmark.name = hg.command.bundle
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.revs = all
# benchmark.variants.type = none-streamv2
before: 316.749699
after: 311.165461 (-1.76%, -5.58)
There is further work to be done in this area like not doing any outgoing
computation in the stream case for example. however the recent changes already
gives use a large win for a small amount of local work.
### benchmark.name = hg.command.bundle
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.revs = all
# benchmark.variants.type = none-streamv2
## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog
pre-%ln-change: 1.263859
the-%ln-change: 0.700229 (-44.60%, -0.56)
prev-changeset: 0.496050 (-60.75%, -0.77)
this-changeset: 0.495243 (-60.81%, -0.77)
## data-env-vars.name = tryton-public-2024-03-22-zstd-sparse-revlog
pre-%ln-change: 2.975765
the-%ln-change: 1.870798 (-37.13%, -1.10)
prev-changeset: 1.461583 (-50.88%, -1.51)
this-changeset: 1.469185 (-50.63%, -1.51)
## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog
pre-%ln-change: 4.540080
the-%ln-change: 3.401700 (-25.07%, -1.14)
prev-changeset: 2.915810 (-35.78%, -1.62)
this-changeset: 2.911643 (-35.87%, -1.63)
## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog
pre-%ln-change: 10.138396
the-%ln-change: 7.750458 (-23.55%, -2.39)
prev-changeset: 6.665565 (-34.25%, -3.47)
this-changeset: 6.672078 (-34.19%, -3.47)
## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
pre-%ln-change: 399.484481
the-%ln-change: 346.508952 (-13.26%, -52.98)
prev-changeset: 316.749699 (-20.71%, -82.73)
this-changeset: 311.165461 (-22.11%, -88.32)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 09 Apr 2024 22:36:35 +0200] rev 51594
outgoing: rework the handling of the `missingroots` case to be faster
The previous implementation was slow, to the point it was taking a significant
amount of `hg bundle --type none-streamv2` call. We rework the code to compute
the same value much faster, making the operation disappear from the `hg bundle
--type none-streamv2` profile. Someone would remark that producing a streamclone
does not requires an `outgoing` object. However that is a matter for another
day. There is other user of `missingroots` (non stream `hg bundle` call for
example), and they will also benefit from this rework.
We implement an old TODO in the process, directly computing the missing and
common attribute as we have most element at hand already.
### benchmark.name = hg.command.bundle
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.revs = all
# benchmark.variants.type = none-streamv2
## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog
before: 7.750458
after: 6.665565 (-14.00%, -1.08)
## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog
before: 0.700229
after: 0.496050 (-29.16%, -0.20)
## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
before: 346.508952
after: 316.749699 (-8.59%, -29.76)
## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog
before: 3.401700
after: 2.915810 (-14.28%, -0.49)
## data-env-vars.name = tryton-public-2024-03-22-zstd-sparse-revlog
before: 1.870798
after: 1.461583 (-21.87%, -0.41)
note: this whole `missingroots` of outgoing has a limited number of callers and
could likely be replace by something simpler (like taking an explicit
"missing_revs" set for example). However this is a wider change and we focus on
a small impact, quick rework that does not change the API for now.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 14 Apr 2024 02:27:10 +0200] rev 51593
proxy-vfs: also proxy the `audit` attribute
In the previous changeset, we had to do a little dance to access the useful
`audit` attribute. We now provide a proper accessors to it.
We don't update the code in `perf.py` because it has to remain compatible with
older version of Mercurial. This will just be nicer in the future.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 13 Apr 2024 23:40:28 +0200] rev 51592
perf: clear vfs audit_cache before each run
When generating a stream clone, we spend a large amount of time auditing path.
Before this changes, the first run was warming the vfs cache for the other
runs, leading to a large runtime difference and a "faulty" reported timing for
the operation.
We now clear this important cache between run to get a more realistic timing.
Below are some example of median time change when clearing these cases. The
maximum time for a run did not changed significantly.
### data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog
# benchmark.name = hg.perf.exchange.stream.generate
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.version = latest
no-clearing: 17.289905
cache-clearing: 21.587965 (+24.86%, +4.30)
## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog
no-clearing: 32.670748
cache-clearing: 40.467095 (+23.86%, +7.80)
## data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
no-clearing: 37.838858
cache-clearing: 46.072749 (+21.76%, +8.23)
## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog
no-clearing: 32.969395
cache-clearing: 39.646209 (+20.25%, +6.68)
In addition, this significantly reduce the timing difference between the
performance command, from the perf extensions and a `real `hg bundle` call
producing a stream bundle. Some significant differences remain especially on
the "mozilla-try" repositories, but they are now smaller.
Note that some of that difference will actually not be
attributable to the stream generation (like maybe phases or branch map
computation).
Below are some benchmarks done on a currently draft changeset fixing some
unrelated slowness in `hg bundle` (34a78972af409d1ff37c29e60f6ca811ad1a457d)
### data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
hg.perf.exchange.stream.generate: 21.587965
hg.command.bundle: 24.301799 (+12.57%, +2.71)
## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog
hg.perf.exchange.stream.generate: 40.467095
hg.command.bundle: 44.831317 (+10.78%, +4.36)
## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog
hg.perf.exchange.stream.generate: 39.646209
hg.command.bundle: 45.395258 (+14.50%, +5.75)
## data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
hg.perf.exchange.stream.generate: 46.072749
hg.command.bundle: 55.882608 (+21.29%, +9.81)
## data-env-vars.name = mozilla-try-2023-03-22-zlib-general-delta
hg.perf.exchange.stream.generate: 334.716708
hg.command.bundle: 377.856767 (+12.89%, +43.14)
## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
hg.perf.exchange.stream.generate: 302.972301
hg.command.bundle: 326.098755 (+7.63%, +23.13)
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 14 Apr 2024 02:41:36 +0200] rev 51591
perf: start recording total time after warming
The warming might be costly and this should not affect the "time profile" of the
actual collection.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 14 Apr 2024 02:40:15 +0200] rev 51590
perf: run the gc before each run
The python garbage collector is a large source of performance troubles, we run
it right before the timed section to reduce the change for the gc to add noise
to the benchmark.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 14 Apr 2024 02:38:41 +0200] rev 51589
perf: allow profiling of more than one run
By default, we still profile the first run only. However profiling more run help
to understand side effect from one run to the other. So we add an option to be
able to do so.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 14 Apr 2024 02:36:55 +0200] rev 51588
profiler: flush after writing the profiler output
Otherwise, the profiler output might only partially appears until the next flush
of the buffer. Since profiling often happens for long operation, the next flush
can be a long time away.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 14 Apr 2024 02:33:36 +0200] rev 51587
stream-clone: disable gc for the entry listing section for the v2 format
This is similar to the change we did for the v3 format in 6e4c8366c5ce.
The benchmark bellow show this gives us a notable gains, especially on larger
repositories.
### benchmark.name = hg.perf.stream-locked-section
# benchmark.name = hg.perf.stream-locked-section
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.version = v2
## data-env-vars.name = pypy-2018-08-01-zstd-sparse-revlog
5e931bf8707c: 0.503820 ~~~~~
1106d1bf695e: 0.470078 (-6.70%, -0.03)
## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog
5e931bf8707c: 0.535756 ~~~~~
1106d1bf695e: 0.490249 (-8.49%, -0.05)
## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog
5e931bf8707c: 1.327041 ~~~~~
1106d1bf695e: 1.174636 (-11.48%, -0.15)
## data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog
5e931bf8707c: 2.439158 ~~~~~
1106d1bf695e: 2.220515 (-8.96%, -0.22)
## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog
5e931bf8707c: 2.630794 ~~~~~
1106d1bf695e: 2.261473 (-14.04%, -0.37)
## data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog
5e931bf8707c: 5.769002 ~~~~~
1106d1bf695e: 5.062000 (-12.26%, -0.71)
## data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
5e931bf8707c: 13.351750 ~~~~~
1106d1bf695e: 12.346655 (-7.53%, -1.01)
## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog
5e931bf8707c: 10.772939 ~~~~~
1106d1bf695e: 9.495407 (-11.86%, -1.28)
## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog
5e931bf8707c: 10.864297 ~~~~~
1106d1bf695e: 9.475597 (-12.78%, -1.39)
## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
5e931bf8707c: 17.448335 ~~~~~
1106d1bf695e: 16.027474 (-8.14%, -1.42)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 09 Apr 2024 02:54:19 +0200] rev 51586
phases: rework the logic of _pushdiscoveryphase to bound complexity
This rework the various graph traversal in _pushdiscoveryphase to keep the
complexity in check.
This is done though a couple of things:
- first, limiting the space we have to explore, for example, if we are not in
publishing push, we don't need to consider remote draft roots that are also
draft locally, as there is nothing to be moved there.
- avoid unbounded descendant computation, and use the faster "rev between"
computation.
This provide a massive boost to performance when exchanging with repository with
a massive amount of draft, like mozilla-try:
### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
# benchmark.name = hg.command.push
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.explicit-rev = all-out-heads
# benchmark.variants.issue6528 = disabled
# benchmark.variants.protocol = ssh
# benchmark.variants.reuse-external-delta-parent = default
## benchmark.variants.revs = any-1-extra-rev
before: 20.346590 seconds
after: 11.232059 seconds (-38.15%, -7.48 seconds)
## benchmark.variants.revs = any-100-extra-rev
before: 24.752051 seconds
after: 15.367412 seconds (-37.91%, -9.38 seconds)
After this changes, the push operation is still quite too slow. Some of this
can be attributed to general phases slowness (reading all the roots from disk
for example) and other know slowness (not using persistent-nodemap, branchmap,
tags, etc. We are also working on them, but with this series, phase discovery
during push no longer showing up in profile and this is a pretty nice and bit
low-hanging fruit out of the way.
### (same case as the above)
# benchmark.variants.revs = any-1-extra-rev
pre-%ln-change: 44.235070
this-changeset: 11.232059 seconds (-74.61%, -33.00 seconds)
# benchmark.variants.revs = any-100-extra-rev
pre-%ln-change: 49.234697
this-changeset: 15.367412 seconds (-68.79%, -33.87 seconds)
Note that with this change, the `hg push` performance is now much closer to the
`hg pull` performance, even it still lagging behind a bit. (and the overall
performance are still too slow).
### data-env-vars.name = mozilla-try-2023-03-22-ds2-pnm
# benchmark.variants.explicit-rev = all-out-heads
# benchmark.variants.issue6528 = disabled
# benchmark.variants.protocol = ssh
# benchmark.variants.pulled-delta-reuse-policy = default
# bin-env-vars.hg.flavor = rust
## benchmark.variants.revs = any-1-extra-rev
hg.command.pull: 6.517450
hg.command.push: 11.219888
## benchmark.variants.revs = any-100-extra-rev
hg.command.pull: 10.160991
hg.command.push: 14.251107
### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.explicit-rev = all-out-heads
# benchmark.variants.issue6528 = disabled
# benchmark.variants.protocol = ssh
# benchmark.variants.pulled-delta-reuse-policy = default
## bin-env-vars.hg.flavor = default
## benchmark.variants.revs = any-1-extra-rev
hg.command.pull: 8.577772
hg.command.push: 11.232059
## bin-env-vars.hg.flavor = default
## benchmark.variants.revs = any-100-extra-rev
hg.command.pull: 13.152976
hg.command.push: 15.367412
## bin-env-vars.hg.flavor = rust
## benchmark.variants.revs = any-1-extra-rev
hg.command.pull: 8.731982
hg.command.push: 11.178751
## bin-env-vars.hg.flavor = rust
## benchmark.variants.revs = any-100-extra-rev
hg.command.pull: 13.184236
hg.command.push: 15.620843
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 05 Apr 2024 22:47:44 +0200] rev 51585
phases: introduce a performant efficient way to access revision in a set
This will be useful in the next changesets.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 05 Apr 2024 14:13:47 +0200] rev 51584
phases: use revision number in `_pushdiscoveryphase`
We now reach our target checkpoint in terms of rev-num conversion. The
`_pushdiscoveryphase` function is now performing graph computation based on
revision number only. Avoiding repeated conversion from node-id to rev-num.
See previous changeset updated `new_heads` for rationnal.
Again, time saved in the 100 milliseconds order of magnitude for the mozilla-try
benchmark I have been using.
However, wow that the logic is done using revision number, we can look into having
better logic in the next changesets, which will provide a much bigger speedup.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 05 Apr 2024 14:11:02 +0200] rev 51583
phases: move RemotePhasesSummary to revision number
This continue our quest to align more logic on revision number instead of
node-ids. The motivation is similar to the change to `new_heads` and
`analyze_remote_phases` a few changeset earlier.
Again, we take this as an opportunity to rename the class, and the attribute to
the new naming scheme. This will highlight the need for code update for any
code using it an expecting node-ids.
Many of the rev-num → node-id conversion we had to introduce in the previous
changesets can now be removed. More will be removed in the future as we continue
to align code toward rev-num usage.
time saved in the 100 milliseconds order of magnitude for the mozilla-try
benchmark I have been using.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 05 Apr 2024 12:24:47 +0200] rev 51582
phases: stop using `repo.set` in `remotephasessummary`
The `repository.set` create changectx on the fly, an expensive operation. Using
`repo.revs` and a direct rev-num → node-id translation will be significantly
faster.
This is especially true as we prepare ourself to no longer do the rev-num →
node-id transalation there.
The speedup is a bit lost in the overall noisyness of the slow phase discovery algorithm, but it save a small amount of time in my benchmark.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 05 Apr 2024 12:02:43 +0200] rev 51581
phases: use revision number in analyze_remote_phases
Same logic as the previous change to `new_heads`, see rationnal there.
This avoids a small number of `nodes -> revs` conversion speeding thing up in
the 100 milliseconds order of magnitude for the worses cases. However, the rest
of the logic is noisy enough that it hardly matters for now.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 05 Apr 2024 11:33:47 +0200] rev 51580
phases: use revision number in new_heads
All graph operations will be done using revision numbers, so passing nodes only
means they will eventually get converted to revision numbers internally.
As part of an effort to align the code on using revision number we make the
`phases.newheads` function operated on revision number, taking them as input
and using them in returns, instead of the node-id it used to consume and
produce.
This is part of multiple changesets effort to translate more part of the logic,
but is done step by step to facilitate the identification of issue that might
arise in mercurial core and extensions.
To make the change simpler to handle for third party extensions, we also rename
the function, using a more modern form. This will help detecting the different
between the node-id version and the rev-num version.
I also take this as an opportunity to add some comment about possible
performance improvement for the future. They don't matter too much now, but they
are worse exploring in a while.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 08 Apr 2024 15:11:49 +0200] rev 51579
phases: convert remote phase root to node while reading them
This is currently a bit silly as we will convert them back to node right after,
but that is an intermediate step before doing more disruptive changes.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 05 Apr 2024 11:17:25 +0200] rev 51578
phases: more compact error handling in analyzeremotephases
using an intermediate variable result in more readable code, so let us use it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 09 Apr 2024 02:54:12 +0200] rev 51577
push: rework the computation of fallbackheads to be correct
The previous computation tried to be smart but ended up being wrong. This was
caught by phase movement test while reworking the phase discovery logic to be
faster.
The previous logic was failing to catch case where the pushed set was not based
on a common heads (i.e. when the discovery seemed to have "over discovered"
content, outside the pushed set)
In the following graph, `e` is a common head and we `hg push -r f`. We need to
detect `c` as a fallback heads and we previous failed to do so::
e
|
d f
|/
c
|
b
|
a
The performance impact of the change seems minimal. On the most impacted
repository at hand (mozilla-try), the slowdown seems mostly mixed in the
overall noise `hg push` but seems to be in the hundred of milliseconds order of
magnitude. When using rust, we seems to be a bit faster, probably because we
leverage more accelaratd internals.
I added a couple of performance related common for further investigation later
on.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 05 Apr 2024 11:05:54 +0200] rev 51576
revset: stop serializing node when using "%ln"
Turning hundred of thousand of node from node to hex and back can be slow… what
about we stop doing it?
In many case were we are using node id we should be using revision id. However
this is not a good reason to have a stupidly slow implementation of "%ln".
This caught my attention again because the phase discovery during push make an
extensive use of "%ln" or huge set. In absolute, that phase discovery probably
should use "%ld" and need to improves its algorithmic complexity, but improving
"%ln" seems simple and long overdue. This greatly speeds up `hg push` on
repository with many drafts.
Here are some relevant poulpe benchmarks:
### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
# benchmark.name = hg.command.push
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.explicit-rev = all-out-heads
# benchmark.variants.issue6528 = disabled
# benchmark.variants.protocol = ssh
# benchmark.variants.reuse-external-delta-parent = default
## benchmark.variants.revs = any-1-extra-rev
before: 44.235070
after: 20.416329 (-53.85%, -23.82)
## benchmark.variants.revs = any-100-extra-rev
before: 49.234697
after: 26.519829 (-46.14%, -22.71)
### benchmark.name = hg.command.bundle
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.revs = all
# benchmark.variants.type = none-streamv2
## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog
before: 10.138396
after: 7.750458 (-23.55%, -2.39)
## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog
before: 1.263859
after: 0.700229 (-44.60%, -0.56)
## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
before: 399.484481
after: 346.5089 (-13.26%, -52.98)
## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog
before: 4.540080
after: 3.401700 (-25.07%, -1.14)
## data-env-vars.name = tryton-public-2024-03-22-zstd-sparse-revlog
before: 2.975765
after: 1.870798 (-37.13%, -1.10)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 09 Apr 2024 14:41:48 +0200] rev 51575
bundlespec: drop unused _bundlespecvariants dictionary
Why do we have a `_bundlespecvariants`?
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 09 Apr 2024 14:37:24 +0200] rev 51574
bundlespec: type the _bundlespeccontentopts dictionary
If only we had a tool to detect the kind of stupid error we just fixed… ho wait.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 09 Apr 2024 14:36:01 +0200] rev 51573
bundlespec: fix the "streamv2" and "streamv3-exp" variant
In c4aab3661f25, we broken this feature by adding unicode instead of bytes to
the dictionary.
On the other hand, this feature was never tested, so augment the tests to tests
this.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Thu, 04 Apr 2024 14:15:32 +0100] rev 51572
wireprotoserver: ensure that output stream gets flushed on exception
Previously flush was happening due to Python finalizer being run on
`BufferedWriter`. With upgrade to Python 3.11 this started randomly
failing.
My guess is that the finalizer on the raw `FileIO` object may
be running before the finalizer of `BufferedWriter` has a chance to run.
At any rate, since we're not relying on finalizers in the happy case
we should also not rely on them in case of exception.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Mon, 15 Apr 2024 16:33:37 +0100] rev 51571
match: strengthen visit_children_set invariant, Recursive means "all files"
My previous interpretation of "Recursive" was too relaxed: I thought it
instructed the caller to do something like this:
> you can stop calling `visit_children_set` because you'll need to descend into
> every directory recursively, but you should still check every file if it
> matches or not
Whereas the real instruction seems to be:
> I guarantee that everything in this subtree matches, you can stop
> querying the matcher for all files and dirs altogether.
The evidence to support this:
- the test actually passes with the stronger invariant, revealing no
exceptions from this rule
- the implementation of `visit_children_set` for `DifferenceMatcher`
clearly relies on this requirement, so it must hold for that not to
lead to bugs.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 12 Apr 2024 16:09:45 +0100] rev 51570
match: fix the rust-side bug in visit_children_set for rootfilesin matchers
The fix is checked by `test_pattern_matcher_visit_children_set` test,
which is what caught the bug in the first place, but also by an end-to-end
test that I made for this purpose.
Accept the new results of Cargo tests
Many of these were already annotated with "FIXME", which is a good sign.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 12 Apr 2024 15:39:21 +0100] rev 51569
match: fix the "visitdir" method on "rootfilesin" matchers
This fixes just the Python side, the fix for the rust side will follow shortly.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 12 Apr 2024 14:21:14 +0100] rev 51568
match: rename RootFiles to RootFilesIn for more consistency
Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 12 Apr 2024 14:17:10 +0100] rev 51567
match: small tweak to PatternMatcher.visit_children_set
This makes it a bit more efficient (avoid a computation in case of early
return), and in my opinion clearer.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 12 Apr 2024 14:09:55 +0100] rev 51566
matchers: fix the bug in rust PatternMatcher that made it cut off early
This brings the rust output in line with the Python output.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 12 Apr 2024 13:48:38 +0100] rev 51565
tests: add an end-to-end test to show a bug in `visit_children_set`
Concretely, `rootfilesin` is completely broken with respect to
`visit_children_set` optimization.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Thu, 11 Apr 2024 19:57:36 +0100] rev 51564
tests: add tests and document expectations from visit_children_set in rust
The tests this patch are adding have the form of formal spec in
invariants::visit_children_set::holds,
and then a series of checks that all examples must satisfy this
formal spec.
I tried to make the spec consistent with how this function is used
and how it was originally conceived. This is in conflict with how it's
documented in Rust. Some of the implementations also fail to implement
this spec, which leads to bugs, in particular when complicated patterns
are used with `hg status`.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Thu, 11 Apr 2024 15:53:23 +0100] rev 51563
tests: add a test that demonstrates a bug in rhg status pattern handling
The bug is in [visit_children_set], will be elaborated on in
follow-up changes.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 05 Apr 2024 01:07:46 +0200] rev 51562
bundle-spec: properly parse boolean configuration as boolean
Before this changesets "v2;revbranchcache=no" would actually request the
addition for a revbranchcache part as the non-empty string `"0"` is `True`
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 04 Apr 2024 16:41:43 +0200] rev 51561
bundle-spec: properly identify changegroup-less bundle
It is possible to produce a bundle without changegroup. For example if we want
to only send phases or obsolescence information. However that lead to crash for
command that identifies bundle content. So we fix that.
The test will come in the next changesets, when we fix another bug preventing to
generate such bundle by hand.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 03 Apr 2024 15:33:25 +0200] rev 51560
perf: create the temporary target next to the source in stream-consume
See inline comment for rational.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 03 Apr 2024 16:00:37 +0200] rev 51559
setup: display return code information about failed `hg` call
This help to understand what is going wrong when things goes wrong.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 02 Apr 2024 21:53:17 +0200] rev 51558
bundlespec: rationalize the way we specify stream bundle version
Instead of having weird dedicated option for each version (v2, v3, etc) we
reuse the same "stream" parameters. This is consistent with the ability to
request a stream clone using "none-v2;stream=v2".
This changeset introduce no user visible change, this is pure internal cleaning.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 02 Apr 2024 17:02:39 +0200] rev 51557
bundle: do no check the changegroup version if no changegroup is included
We don't need to check the compatibility of something we will not use.
In practice this was getting in the was of `streamv2` bundles on a narrow
repository as the 'cg.version=02' value was rejected by this checks.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 27 Mar 2024 18:51:33 +0000] rev 51556
perf-stream-consume: use the source repository config when applying
This might contains critical configuration for the benchmark, like enabling of
extensions like narrow.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 27 Mar 2024 17:46:23 +0000] rev 51555
unbundle: move most of the logic on cmdutil to help debug::unbundle reuse
This make sure `hg debug::unbundle` focus on the core logic.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 27 Mar 2024 17:29:48 +0000] rev 51554
postincoming: move to cmdutil
This looks like a good place for it to live.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 27 Mar 2024 17:21:46 +0000] rev 51553
postincoming: avoid computing branchhead if no report will be posted
This otherwise defeat some of the branch v3 optimization.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 26 Mar 2024 13:46:44 +0000] rev 51552
streamclone: stop listing files for entries that have no volatile files
This will save a lot of python related time.
This significantly boost performance. The following number comes from a large
private repository using perf::stream-locked-section:
base-line: 35.04 seconds
prev-change: 24.51 seconds (-30%)
prev-change: 20.88 seconds (-40%)
prev-change: 14.22 seconds (-60%)
this-change: 11.58 seconds (-67% from baseline; -18% from prev)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 26 Mar 2024 13:34:05 +0000] rev 51551
stream-clone: disable gc for the initial section for the v3 format
The number of small container created turn Python in a gc-frenzy that seriously
impact performance.
This significantly boost performance. The following number comes from a large
private repository using perf::stream-locked-section:
base-line: 35.04 seconds
prev-change: 24.51 seconds (-30%)
prev-change: 20.88 seconds (-40%)
this-change: 14.22 seconds (-60% from baseline; -31% from prev)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 26 Mar 2024 13:32:46 +0000] rev 51550
stream-clone: disable gc for `_entries_walk` duration
The number of small container created turn Python in a gc-frenzy that seriously
impact performance.
This significantly boost performance. The following number comes from a large
private repository using perf::stream-locked-section:
base-line: 35.04 seconds
prev-change: 24.51 seconds (-30%)
this-change: 20.88 seconds (-40% from baseline; -15% from previous changes)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 26 Mar 2024 13:28:52 +0000] rev 51549
nocg: make the utility work are both a decorator and context manager
In some case, the context manager version will be simpler.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 26 Mar 2024 11:24:20 +0000] rev 51548
stream-clone: stop getting the file size of all file in v3
The point of v3 is to do less work in the locked section. It was currently not
the case.
This significantly boost performance. The following number comes from a large
private repository using perf::stream-locked-section:
base-line: 35.03 seconds
this-change: 24.50 seconds (-30%)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 26 Mar 2024 18:55:40 +0000] rev 51547
stream: in v3, skip the "size" fast path if the entries as some unknown size
We are about to prefetch size during the lock less in the v3 case. So we need to
avoid trying to use that prefetched size when it is not available.
See next changeset for the motivation.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 26 Mar 2024 08:43:20 +0000] rev 51546
perf-stream-locked-section: advertise the right version key in the help
As the v3 format is still experimental, its key is "v3-exp". The help text was
not pointing that out.
(we also fix `perf::stream-generate` in the process)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 26 Mar 2024 08:39:08 +0000] rev 51545
perf-stream-locked-section: fix the call to the v3 generator
That generator simply return chunks so we should not assign the return to a
tuple.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 26 Mar 2024 08:36:47 +0000] rev 51544
perf-stream-locked-section: actually use v1 generation when requested
We were fetching a v1 generator but actually using the v2 function…
Raphaël Gomès <rgomes@octobus.net> [Fri, 29 Mar 2024 21:39:00 +0100] rev 51543
branching: merge stable into default
Raphaël Gomès <rgomes@octobus.net> [Fri, 29 Mar 2024 21:37:09 +0100] rev 51542
Added signature for changeset 803e61387e86
Raphaël Gomès <rgomes@octobus.net> [Fri, 29 Mar 2024 21:37:06 +0100] rev 51541
Added tag 6.7.2 for changeset 803e61387e86
Raphaël Gomès <rgomes@octobus.net> [Thu, 28 Mar 2024 14:47:20 +0000] rev 51540
relnotes: add 6.7.2
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 28 Mar 2024 07:12:09 +0000] rev 51539
bundle2: make the "hgtagsfnodes" part advisory
This bundle2 part is about helping the client to warms its cache. There is no
reason for it to be mandatory.
So we mark it advisory.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 25 Mar 2024 16:27:48 +0000] rev 51538
branching: merge stable into default
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 07 Mar 2024 10:57:16 +0100] rev 51537
branchcache: allow to detect "pure topological case" for branchmap
We don't rum this detection every time we run the branchcache, that would be
costly. However we now do it when running `hg debugupdatecache`.
This will help existing repository to benefit from the fastpath when possible.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 07 Mar 2024 04:15:23 +0100] rev 51536
branchcache: add a "pure topological head" fast path
In a narrow but actually quick common case, all topological heads are all on
the same branch and all open. In this case, computing the branch map is very
simple. We can quickly detect situation where this situation will not change.
So we update the V3 format to be able to express this situation and upgrade the
update code to detect we remains in that mode.
The branch cache is populated with the actual value when the branch map is
accessed, but the update_disk method can do the update without needing to
populate it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 06 Mar 2024 16:18:03 +0100] rev 51535
branchcache: move the processing of the new data in a dedicated method
In a future changeset, this will allow the V3 of the branch cache to use a fast
path when possible.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 06 Mar 2024 16:10:44 +0100] rev 51534
branchcache: gather newly closed head in a dedicated set
This is part of a series to more clearly split the update in two step. This
will allow us to introduce a fast path during update in a future changeset.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 06 Mar 2024 16:09:42 +0100] rev 51533
branchcache: gather new obsolete revision in a set
This is part of a series to more clearly split the update in two step. This
will allow us to introduce a fast path during update in a future changeset.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 06 Mar 2024 15:54:22 +0100] rev 51532
branchcache: filter obsolete revisions sooner
Since we won't do anything with the obsolete revisions, we can just ignore them
sooner.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 07 Mar 2024 10:55:22 +0100] rev 51531
branchcache: skip entries that are topological heads in the on disk file
In the majority of cases, topological heads are also branch heads. We have
efficient way to get the topological heads and efficient way to retrieve
their branch information. So there is little value in putting them in the branch
cache file explicitly. On the contrary, writing them explicitly tend to create
very large cache file that are inefficient to read and update.
So the branch cache v3 format is no longer including them. This changeset focus
on the format aspect and have no focus on the performance aspect. We will cover
that later.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 07 Mar 2024 01:35:43 +0100] rev 51530
branchcache: simplify the branch rev cache test
We don't need that many content dump and this gets in the way in change in
access pattern (e.g. accessing revision in a different order change the order of
branches in the "names" file).
So we simplify this test in advance.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 06 Mar 2024 11:39:44 +0100] rev 51529
branchcache: store filtered hash and obsolete hash independently for V3
This will avoid the bug covered in tests/test-branches-obsolete.t when we stop
storing all heads explicitly in V3.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 06 Mar 2024 12:07:31 +0100] rev 51528
branchcache: show the cache file content in test-branches-obsoletes.t
This help to track the changes in format between v2 and v3.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 06 Mar 2024 02:20:53 +0100] rev 51527
branchcache: rework the `filteredhash` logic to be more generic
We now have a more flexible `key_hashes` tuple. We duplicated various logic in
the V2 and V3 version of the cache as the goal is to start changing the logic
for V3 in the next few changesets.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 06 Mar 2024 01:53:52 +0100] rev 51526
filteredhash: rename the filteredhash function
The new name is less ambiguous, as we are about to introduce an alternative
function it seems like a good idea to have clearer name to distinct the two.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 06 Mar 2024 01:43:51 +0100] rev 51525
filteredhash: split the computation of revision sets
The branch2's filteredhash combines the filtered revisions and the obsolete
ones, this will creates issue for implicit reference to heads we want to
introduce for the v3 of the branch cache format. So we isolate this logic for
alternative use.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 05 Mar 2024 15:21:18 +0100] rev 51524
filteredhash: move the hashing in its own function
This will help us to reuse this logic in variants of the hashes used for branch
cache validation.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 25 Feb 2024 23:31:50 +0100] rev 51523
branchcache: cleanup the final key generation after update
A lot of duplicated work seemed to be done, as we already update the tiprev and
tipnode when needed right before. So we simplify that part to focus on the
filtered hash.
See inline comment for details.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 28 Feb 2024 12:56:08 +0100] rev 51522
branchcache: add more test for the logic around obsolescence and branch heads
While working on branch-cache-v3, we noticed some ambiguity in the
filtered+obsolete hash. However this was only caught by a rebase test by
chance.
It seems important to explicitly tests these cases.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 26 Feb 2024 15:44:44 +0100] rev 51521
branchcache-v3: use more explicit header line
The key-value approach is clearer and gives more rooms to have the format evolve
in a clear way. It also provides extension (like topic) simpler way to extend
the validation scheme.
This is just a small evolution, the V3 format is still a work in progress.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 26 Feb 2024 14:20:36 +0100] rev 51520
branchcache-v3: introduce a v3 format
For now the format is the very same, however we will start changing it in
future changesets.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 27 Feb 2024 14:04:29 +0100] rev 51519
branchcache: use an explicit class for the v2 version
This prepare the introduction of an experimental v3 format version.
In the process, we move the description of the format in that new class.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 27 Feb 2024 15:33:21 +0100] rev 51518
branchcache: add some blank line in a test
This helps each section to stand out.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 25 Mar 2024 02:09:15 +0100] rev 51517
phases: update the phase set as we go during retract boundary
Apparently iterating over the `changed_revs` dictionary is very expensive.
On mozilla-try-2019-02-18, a perf::unbundle call with a 10 000 changesets
bundle gives give use the following timing.
e57d4b868a3e: 4.6 seconds
ac1c75188440: 102.5 seconds
prev-changeset: 30.0 seconds
this-changeset: 4.6 seconds
So, the performance regression is gone.
Once again: thanks to marvelous Python!
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 25 Mar 2024 01:50:31 +0100] rev 51516
phases: avoid a potentially costly dictionary interation in some case
If we retract for the draft phase, there is not non-public item to be retracted
and we can skip this part. This part is was apparently super costly thanks to
Python.
On mozilla-try-2019-02-18, a perf::unbundle call with a 10 000 changesets
bundle gives give use the following timing.
e57d4b868a3e: 4.6 seconds
ac1c75188440: 102.5 seconds
this-changeset: 30.0 seconds
So we recovered about ⅔ of the regression, the next changeset will give us the
rest back.
Raphaël Gomès <rgomes@octobus.net> [Thu, 21 Mar 2024 12:26:46 +0100] rev 51515
branching: merge stable into default
Raphaël Gomès <rgomes@octobus.net> [Thu, 21 Mar 2024 12:24:42 +0100] rev 51514
Added signature for changeset 2e6fde2ed01e
Raphaël Gomès <rgomes@octobus.net> [Thu, 21 Mar 2024 12:24:36 +0100] rev 51513
Added tag 6.7.1 for changeset 2e6fde2ed01e
Raphaël Gomès <rgomes@octobus.net> [Thu, 21 Mar 2024 12:23:25 +0100] rev 51512
relnotes: add 6.7.1
Felipe Resende <felipe@fcresende.dev.br> [Sat, 16 Mar 2024 21:02:19 -0300] rev 51511
subrepo: fix normalizing paths with scheme
After revision 0afe96e374a7, subrepo paths were normalized using
posixpath.normpath and that resulted in ssh paths being wrongly converted
from ssh://host/path to ssh:/host/path
This fix applies the same logic used in urlutil.url to split the path scheme
from the rest and only use posixpath.normpath to the string after scheme://
Felipe Resende <felipe@fcresende.dev.br> [Sat, 16 Mar 2024 18:37:07 -0300] rev 51510
sshpeer: fix path when handling invalid url exception
In 73ed1d13c0bf the code was refactored but the error handling seems to have
been missed (or maybe the object shoud have implemented __bytes__)
Raphaël Gomès <rgomes@octobus.net> [Mon, 18 Mar 2024 11:25:21 +0100] rev 51509
delta-search: fix crash caused by unbound variable
This code path was apparently not tested. This fixes a crash when cloning the
Tryton repo.
Raphaël Gomès <rgomes@octobus.net> [Fri, 15 Mar 2024 10:52:51 +0100] rev 51508
branching: merge stable into default
Raphaël Gomès <rgomes@octobus.net> [Fri, 15 Mar 2024 10:49:44 +0100] rev 51507
Added signature for changeset c9ceb4f60256