rust-nodemap: accounting for dead blocks By the very append-only nature of the `NodeTree`, inserting new blocks has the effect of making some of the older ones useless as they become unreachable. Therefore some automatic housekeeping will need to be provided. This is standard procedure in the word of databases, under names such as "repack" or "vacuum". The new `masked_readonly_blocks()` will provide callers with useful information to decide if the nodetree is ripe for repacking, but all the `NodeTree` can provide is how many blocks have been masked in the currently mutable part. Analysing the readonly part would be way too long to do it for each transaction and defeat the whole purpose of nodemap persistence. Serializing callers (from the Python layer) will get this figure before each extraction and maintain an aggregate counter of unreachable blocks separately. Note: at this point, the most efficient repacking is just to restart afresh with a full rescan. Differential Revision: https://phab.mercurial-scm.org/D8097

rust-nodemap: core implementation for shortest In this implementation, we just make `lookup()` return also the number of steps that have been needed to come to a conclusion from the nodetree data, and `validate_candidate()` takes care of the special cases related to `NULL_NODE`. This way of doing minimizes code duplication, but it means that the comparatively slower finding of first non zero nybble will run for all calls to `find()` where it is not needed. Still running on the file generated for the mozilla-central repository, it seems indeed that we now get more ofter 320 ns than 310. The odds that this could have a significant impact on real life Mercurial performance are still looking low. Let's wait for actual benchmark runs to see if an optimization is needed here. Differential Revision: https://phab.mercurial-scm.org/D7819

rust-nodemap: special case for prefixes of NULL_NODE We have to behave as though NULL_NODE was stored in the node tree, although we don't store it. Differential Revision: https://phab.mercurial-scm.org/D7798

rust-nodemap: pure Rust example To run, use `cargo run --release --example nodemap` This demonstrates that simple scenarios entirely written in Rust can content themselves with `NodeTree<T>`. The example mmaps both the nodemap file and the changelog index. We had of course to include an implementation of `RevlogIndex` directly, which isn't much at this stage. It felt a bit prematurate to include it in the lib. Here are some first performance measurements, obtained with this example, on a clone of mozilla-central with 440000 changesets: (create) Nodemap constructed in RAM in 153.638305ms (query CAE63161B68962) found in 22.362us: Ok(Some(269489)) (bench) Did 3 queries in 36.418µs (mean 12.139µs) (bench) Did 50 queries in 184.318µs (mean 3.686µs) (bench) Did 100000 queries in 31.053461ms (mean 310ns) To be fair, even between bench runs, results tend to depend whether the file is still in kernel caches, and it's not so easy to get back to a real cold start. The worst we've seen was in the 50us ballpark. In any busy server setting, the pages would always be in RAM. We hope it's good enough not to be significantly slower on any concrete Mercurial operation than the C nodetree when fully in RAM, and of course this implementation has the serious headstart advantage of persistence. Differential Revision: https://phab.mercurial-scm.org/D7797

rust-nodemap: input/output primitives These allow to initiate a `NodeTree` from an immutable opaque sequence of bytes, which could be passed over from Python (extracted from a `PyBuffer`) or directly mmapped from a file. Conversely, we can consume a `NodeTree`, extracting the bytes that express what has been added to the immutable part, together with the original immutable part. This gives callers the choice to start a new Nodetree. After writing to disk, some would prefer to reread for best guarantees (very cheap if mmapping), some others will find it more convenient to grow the memory that was considered immutable in the `NodeTree` and continue from there. This is enough to build examples running on real data and start gathering performance hints. Differential Revision: https://phab.mercurial-scm.org/D7796

pyoxidizer: allow extensions to be loaded from the file system It seems that setting this config is all that's needed to be able to load extensions from the file system (which we clearly want). Thanks for making this work, Gregory Szorc!. Differential Revision: https://phab.mercurial-scm.org/D8122

graft: always allow hg graft --base . (issue6248) `hg graft --base . -r abc` is rejected before this change with a "nothing to merge" error, if `abc` does not descend from `.`. This looks like an artifact of the implementation rather than intended behavior. It makes perfect sense to apply the diff between `.` and `abc` to the working copy (i.e. degenerate into `hg revert`), regardless of what `abc` is. Differential Revision: https://phab.mercurial-scm.org/D8127

revlog-compression: update the config to be a list format.revlog-compression is now a list of engine, the first supported one is to be used. Doing this have several benefits: 1) this is fully backward compatible, config using a single entry will be read as a single item list, not changing any behavior. 2) This open the way to use zstd by default without impacting platform were it is not available. This will be done in a later changesets. Using zstd provide a significant performance boost explained in : bb271ec2fbfb. However zstd is not available in some cases, A notable example is the `--pure` version of Mercurial which doesn't come with zstd support. Differential Revision: https://phab.mercurial-scm.org/D8148

remotefilelog: add 'changelog' arg to shallowcg1packer.generate (issue6269) This cause traceback on widening using narrow extension when remotefilelog is enabled. Differential Revision: https://phab.mercurial-scm.org/D8134

drawdag: abide by new createmarkers() API The `obsolete.createmarkers()` API was changed in 6335c0de80fa (obsolete: allow multiple predecessors in createmarkers, 2018-09-22) to prefer its precursors input to be a tuple instead of a single precursor. Let's fix `drawdag.py` to comply. Differential Revision: https://phab.mercurial-scm.org/D8149