delta-find: remove dead code intended to deal with forced delta reuse Since the case was dealt with sooner (see XXX), we no longer need to deal with it in this part of the code.

library: incorporate demandimport into runpy invocation

library: enable runpy invocation on mercurial package

match: add `filepath:` pattern to match an exact filepath relative to the root It's useful in certain automated workflows to make sure we recurse in directories whose name conflicts with files in other revisions. In addition it makes it possible to avoid building a potentially costly regex, improving performance when the set of files to match explicitly is large. The benchmark below are run in the following configuration : # data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog # benchmark.name = files # benchmark.variants.rev = tip # benchmark.variants.files = all-list-filepath-sorted # bin-env-vars.hg.flavor = no-rust It also includes timings using the re2 engine (through the `google-re2` module) to show how much can be saved by just using a better regexp engine. Pattern time (seconds) time using re2 ----------------------------------------------------------- just "." 0.4 0.4 list of "filepath:…" 1.3 1.3 list of "path:…" 25.7 3.9 list of patterns 29.7 10.4 As you can see, Without re2, using "filepath:" instead of "path:" is a huge win. With re2, it is still about three times faster to not have to build the regex.

path: use the next `display` argument to deal with boolean We have a generic mechanism that we can now use for the special case now.

path: display proper user facing value for pulled-delta-reuse-policy The integer was not what the user expect nor what could be provided as a config value.

paths: add an argument to format the suboption display We will use it in the next function to the delta policy display. It could also be use to deal with the other special case in the command code, but that is unnecessary churn for stable so that part will go on default.

ui: fix crash in `hg paths` when pulled-delta-reuse-policy is in use The crash is a result of [value] being an int, but being used as a bytestring.

ui: add a test demonstrating a crash in `hg paths`

re2: fix reporting of availability in `hg debuginstall` We add and use an official API to check is re2 is available. This prevent the bug previously in place were debuginstall was considering confusing `None` (i.e. non-initialized) and `False` (i.e. unavailable).

re: make _checkre2 a static method There is no need to it to be a method, so we make it at least a static method to make it easier to call.

re2: exit `_checkre2` early if calling it is useless If we already know if re2 is supported there is no need to run the check again. Stricly speaking we don't call it twice, but this seems a simple enough safeguard to include in this series.

branching: merge with stable

backout: remove db9b6ce803a9 That changeset lost its topic for unknown reason and ended up being picked by some automation. So I backout it previous incarnation until it can get properly reviewed.

infinitepush: aggressively deprecated infinite push We add a large warning and an explicit abort. Both can be disable through config, but neither are explicitly documented. If any user exists I really want them to get in touch with the community.

rust-dependencies: switch from `users` to `whoami` `users` is unmaintained, with the maintainer apparently MIA. `whoami` is a popular crate that does simple wrapping of platform-specific calls. This makes the overhead from using `blackbox` lower. It used to be up 10ms on slower hardware for both calls, now down to <1ms which is always good.

rust-hg-core: move from `ouroboros` to `self_cell` `ouroboros` has a fundamental soundness problem that, while not applicable today, could become applicable given new compiler optimizations.¹ `self_cell` is a crate that accomplishes a lot of the same things that `ouroboros` did while remaining sound (that is, unless a new soundness issue is discovered) by not assuming as much about the memory layout of the program. `self_cell` has been scrutinized heavily in the past few months by very competent people, some from the compiler team and has shown no weaknesses for a while, with a 1.0 stable release coming out a couple months ago. Our internal API is exactly the same, this is just an implementation detail. To reiterate, no actual soundness issue was found with our use of `ouroboros`, but there might be evolutions of `rustc` (or even a future separate compiler) that could generate unsound code. [1] https://github.com/joshua-maros/ouroboros/issues/88

debug-revlog-stats: make it use the new store entry API This give use free tree manifest support in the process.

store: use the revlog type from revlog's constant instead of custom one We already have a revlog type that everybody use. So we use that one instead.

repair: use `is_manifestlog` to recognise manifest revlog This is simpler that comparing flag, and we want to remove these flags anyway.

perf: add a perf::stream-consume We know how long it take to generate, lets check how long it take to apply now.

perf: add a perf::stream-generate command This record the time we take to generate a bundle.

perf: add a new "context" argument to timer This allow to simple setup/teardown outside of the timed section. Especially using object that need context manager, like a temporary files.

perf: add support for stream-v3 during benchmark This is getting important as the v3 protocol will diverge from the v2 protocol.

perf: add a function to find a stream version generator The logic is clearer and can be reused for other commands in the future.

treemanifest: make `updatecaches` update the nodemaps for all directories Without this, if the cache for a nested directory is in a bad state, it's very hard to repair it.

stream-clone: avoid opening a revlog in case we do not need it Opening an revlog has a cost, especially if it is inline as we have to scan the file and construct an index. To prevent the associated slowdown, we just do a minimal scan to check that an inline file is still inline, and simply stream the file without creating a revlog when we can. This provides a big boost compared to the previous changeset, even if the full generation is still penalized by the initial gathering of information. All benchmarks are run on linux with Python 3.10.7. # benchmark.name = hg.exchange.stream.generate # benchmark.variants.version = v2 ### Compared to the previous changesets We get a large win all across the board! # mercurial-2018-08-01-zstd-sparse-revlog before: 0.250694 seconds after: 0.105986 seconds (-57.72%) # pypy-2018-08-01-zstd-sparse-revlog before: 3.885657 seconds after: 1.709748 seconds (-56.00%) # netbeans-2018-08-01-zstd-sparse-revlog before: 16.679371 seconds after: 7.687469 seconds (-53.91%) # mozilla-central-2018-08-01-zstd-sparse-revlog before: 38.575482 seconds after: 17.520316 seconds (-54.58%) # mozilla-try-2019-02-18-zstd-sparse-revlog before: 81.160994 seconds after: 37.073753 seconds (-54.32%) ### Compared to 6.4.3 We are still significantly slower than 6.4.3, the extra time is usually twice slower than the extra time we observe on the locked section, which is a quite interesting information. Except for mercurial-central that is much faster. That discrepancy is not really explained yet. # mercurial-2018-08-01-zstd-sparse-revlog 6.4.3: 0.072560 seconds after: 0.105986 seconds (+46.07%) (- 0.03 seconds) # pypy-2018-08-01-zstd-sparse-revlog 6.4.3: 1.211193 seconds after: 1.709748 seconds (+41.16%) (-0.45 seconds) # netbeans-2018-08-01-zstd-sparse-revlog 6.4.3: 4.932843 seconds after: 7.687469 seconds (+55.84%) (-2.75 seconds) # mozilla-central-2018-08-01-zstd-sparse-revlog 6.4.3: 34.012226 seconds after: 17.520316 seconds (-48.49%) (-16.49 seconds) # mozilla-try-2019-02-18-zstd-sparse-revlog 6.4.3: 23.850555 seconds after: 37.073753 seconds (+55.44%) (+13.22 seconds)

store: stop relying on a `revlog_type` property We want to know if a file is related to a revlog, but the rest is dealt with differently already, so we simplify things further. as a bonus, this cleanup This provides a small but noticeable speedup. The number below use `hg perf::stream-locked-section` to measure the time spend in the locked section of the streaming clone. Number are run on various repository and compare different steps.: 1) the effect of this patchs, 2) the effect of the cleanup series, 2) current state compared to because large refactoring. All benchmarks are run on linux with Python 3.10.7. ### Effect of this patch # mercurial-2018-08-01-zstd-sparse-revlog # benchmark.name = perf-stream-locked-section before: 0.030246 seconds after: 0.029274 seconds (-3.21%) # pypy-2018-08-01-zstd-sparse-revlog before: 0.545012 seconds after: 0.520872 seconds (-4.43%) # netbeans-2018-08-01-zstd-sparse-revlog before: 2.719939 seconds after: 2.626791 seconds (-3.42%) # mozilla-central-2018-08-01-zstd-sparse-revlog before: 6.304179 seconds after: 6.096700 seconds (-3.29%) # mozilla-try-2019-02-18-zstd-sparse-revlog before: 14.142687 seconds after: 13.640779 seconds (-3.55%) ### Effect of this series A small but sizeable speedup # mercurial-2018-08-01-zstd-sparse-revlog before: 0.031122 seconds after: 0.029274 seconds (-5.94%) # pypy-2018-08-01-zstd-sparse-revlog before: 0.589970 seconds after: 0.520872 seconds (-11.71%) # netbeans-2018-08-01-zstd-sparse-revlog before: 2.980300 seconds after: 2.626791 seconds (-11.86%) # mozilla-central-2018-08-01-zstd-sparse-revlog before: 6.863204 seconds after: 6.096700 seconds (-11.17%) # mozilla-try-2019-02-18-zstd-sparse-revlog before: 14.921393 seconds after: 13.640779 seconds (-8.58%) ### Current state compared to the pre-refactoring state The refactoring introduced multiple string manipulation and dictionary creation that seems to induce a signifiant slowdown Slowdown # mercurial-2018-08-01-zstd-sparse-revlog 6.4.3: 0.019459 seconds after: 0.029274 seconds (+50.44%) ## pypy-2018-08-01-zstd-sparse-revlog 6.4.3: 0.290715 seconds after: 0.520872 seconds (+79.17%) # netbeans-2018-08-01-zstd-sparse-revlog 6.4.3: 1.403447 seconds after: 2.626791 seconds (+87.17%) # mozilla-central-2018-08-01-zstd-sparse-revlog 6.4.3: 3.163549 seconds after: 6.096700 seconds (+92.72%) # mozilla-try-2019-02-18-zstd-sparse-revlog 6.4.3: 6.702184 seconds after: 13.640779 seconds (+103.53%)

store: directly pass the filesize in the `details` of revlog The dictionary only contains 1 (or 0) entries, we can directly store that information (or None). Moving to a simpler argument passing result in a noticable speedup (because Python) The number below use `hg perf::stream-locked-section` to measure the time spend in the locked section of the streaming clone. Number are run on various repository. ### mercurial-2018-08-01-zstd-sparse-revlog before: 0.031247 seconds after: 0.030246 seconds (-3.20%) ### mozilla-central-2018-08-01-zstd-sparse-revlog before: 6.718968 seconds after: 6.304179 seconds (-6.17%) ### mozilla-try-2019-02-18-zstd-sparse-revlog before: 14.631343 seconds after: 14.142687 seconds (-3.34%) ### netbeans-2018-08-01-zstd-sparse-revlog before: 2.895584 seconds after: 2.719939 seconds (-6.07%) ### pypy-2018-08-01-zstd-sparse-revlog before: 0.561843 seconds after: 0.543034 seconds (-3.35%)

store: explicitly pass file_size when creating StoreFile A small cleanup before large cleanup in the next patch.

store: have the revlog determine which files are volatile itself This is a first step toward simplifying the walk step.

test-dirstate-version-fallback: future-proof the test for a different default Dirstate-v2 will become the default at some point, which would cause this test to fail. Let's save someone else the headache later.

clonebundles: add support for inline (streaming) clonebundles The idea behind inline clonebundles is to send them through the ssh or https connection to the Mercurial server. We've been using this specifically for streaming clonebundles, although it works for 'regular' clonebundles as well (but is less relevant, since pullbundles exist). We've had this enabled for around 9 months for a part of our users. A few benefits are: - no need to secure an external system, since everything goes through the same Mercurial server - easier scaling (in our case: no risk of inconsistencies between multiple mercurial-server mirrors and nginx clonebundles hosts) Remaining topics/questions right now: - The inline clonebundles don't work for https yet. This is because httppeer doesn't seem to support sending client capabilities. I didn't focus on that as my main goal was to get this working for ssh.

Added signature for changeset da372c745e0f

Added tag 6.4.4 for changeset da372c745e0f

relnotes: add 6.4.4

revlog: avoid possible collision between directory and temporary index Since 6.4, we create a temporary index file to write the split data without overwriting the inline version too early. However, the store encoding does not prevent these new `.i.s` file to collide with a directory with the same name. While the odds for such a collision to happens are fairly low, the collision would prevent Mercurial from working. The store encoding have a mitigation solution in place to prevent such collisions from happening for `.i` and `.d` files, but not for other extensions. We cannot update this encoding scheme to solve the issue since it would diverge from older version of Mercurial. Instead, we create an alternative directory tree dedicated to such files. The use of the `.i` extension combined with store encoding will prevent collisions there.

revlog: move the computation of the split_index path in a property This is about to become more complex, so we gather the logic in a single place.

rust-dirstate: fall back to v1 if reading v2 failed This will help us not fail when a v1 dirstate is present on disk while a v2 was expected (which could happen with a racy/interrupted upgrade).

dirstate: add test showing dirstate version mismatch causes an error We should fall back to trying dirstate v1 when v2 fails to read.

rust-dirstate: rename `has_dirstate_v2` to `use_dirstate_v2` It is closer to the right semantics. I added a docstring to better explain the reasonning. In the next patch(es), I will address the underlying issue of finding the "wrong" version of the dirstate on disk.

delta-find: fix pulled-delta-reuse-policy=forced behavior The code that select delta still has too many oportunity to discard the delta is has been forcibly asked to reuse. However is is fairly easy to use a dedicated fastpath for this case. So we do so. Cleaning other code that tries to enforce that policy will be done on default.

delta-find: display more information about the search in some case This will be useful to access the effect of the delta reuse policy.

deltafind: issue debug information when we fast-path rivial case too More debug options never hurts.

delta-find: gather the condition to blindly use a full snapshot together We are about to make the `if` body bigger, so having only one of them is simpler/

delta-find: initialize the debug information much sooner (when possible) This help us to record debug information in alternative path.

delta-find: fix `parents` round detection We should compare integer with integer, instead of bytes (node).

delta-find: intrduce a `_one_dbg_data` method This helps with the initialisation of the expected debug information.

delta-find: move final debug processing in a `_dbg_process_data` method This will help us making sure we can call the necessary debugging code in alternative codepath.

delta-fine: use the `_debug_search` attribute directly By adjusting the value at creation time, we can avoid the need to use and intermediate variable.

delta-find: move the `gather_debug` logic in a property This will make it simpler to get be reused in multiple places.

tests: check `pulled-delta-reuse-policy=forced` without general delta If general delta is not available, the amount of delta we can reuse at all will be more limited. We check that application still work and does not corrupt the repository.

tree-manifest: allow `debugupgraderepo` to run on tree manifest repo There does not seems to be anything wrong with running the current logic on them. So we remove the limitation.

bundles: clarify streaming v2 bundle usage This seems useful to clarify through the documentation (we were still using debugcreatestreamclonebundle, even though 'hg bundle' is already an option...).

stream-clone: update debugcreatestreamclonebundle helps People to stop using streamv1, so we should point them to alternative in the place where people might find it.

rewrite: simplify the `retained_extras` extra logic First, we move the definition of value outside of the rebase extensions, as this apply to all rebase-like operation and some live in other place (like evolve). Second we make it a simple set, so that it is easy for an extension to add a new value in it. Third, we move the associated logic in core too. That make it easily available to other extensions. Fourth we simplify it usage, as the verbose version of the filtering is just a handful on line long, we are just going to test all the value for updates, so the Projection overlay is not bringing much here. Note that, we make it a module level set, is a key is worth preserving it is probably worth preserving in all cases. This was already the behavior prior to this change.

revlog: fix a bug in revlog splitting Specifically, if the .i.s files are not added to the transaction, then a follow-up modification of such a file records it into transaction as non-empty, which is incorrect.

stream-clone: smoothly detect and handle a case were a revlog is split This detect and handle the most common case for a race condition around stream and revlog splitting. The one were the revlog is split between the initial collection of data and the time were we start considering stream that data. In such case, we repatch an inlined version of that revlog together when this happens. This is necessary as stream-v2 promised a specific number of bytes and a specific number of files to the client. In stream-v3, we will have the opportunity to just send a split revlog instead. Getting a better version of the protocol for stream-v3 is still useful, but it is no longer a blocket to fix that race condition. Note that another, rarer race condition exist, were the revlog is split while we creating the revlog and extracing content from it. This can be dealt with later.

stream-clone: implement decidated `get_streams` method for revlog For revlog, we can do better using the maximum linkrev expected. This approach open the way to dealing with a much larger set of non-trivial changes, like splitting of inline revlogs. We will actually tackle this issue in the next changesets (thanks to this one).

stream-clone: make it the responsability of the store entry to stream content The store entry has more context, this will especially be true when it comes to revlogs. So we move the details of how to retrieve binary content to the StoreEntry. The stream clone code now focus on the protocol bits.