upgrade: correct implementation of improvement.__ne__ The "not" operator binds more closely than "==": >>> not False == False False

extensions: use ui.log() interface to provide detailed loading information The output format changes and the messages will be sent to stderr instead of stdout, but I don't think that matters.

mq: implement log() on dummyui Otherwise ui.log() in extensions.py would explode.

ui: install logger that sends debug.extensions messages to stderr This will replace the custom log function introduced at d58958676b3c "extensions: add detailed loading information."

blackbox: resurrect recursion guard If I added ui.log() to hg.repository() function, test-merge-subrepos.t exploded. The problem is that the blackbox may create new repository instance while logging is active, and the created repository owns its new ui derived from the baseui, not from the ui which is processing the active logging. I tried to work around the issue in ui.log(), but that turned out to be not easy. We shouldn't globally lock the ui.log() since there may be more than one active repo/ui instances in threaded environment. We could store the logging state in thread-local storage, but that seems unnecessarily complex. So this patch reintroduces the _inlog flag to per-repository logger instances.

tests: filter out uninteresting log events This helps adding more log()s without updating the tests.

context: error out if basefilectx.cmp() is called without self._filenode The base implementation can't handle such cases because the filelog has no knowledge about the working directory. Loading self._filenode should have no extra cost since self.size() would load it anyway.

context: collapse complex condition to see if filelog have to be compared It's hard to read. I'd rather make the return statement duplicated.

merge with stable

worker: do not swallow exception occurred in main process Before, SystemExit(255) would be most likely raised since the worker processes were terminated by the main process and the status would be set to 255 in response. We should instead re-raise the exception occurred first. It's pretty hard to debug problems like the issue 6035 with no traceback.

run-tests: fix permission to clean up unreadable directories I found many hgtests.* directories left in $TMPDIR, which couldn't be deleted because test-ssh-repoerror.t created some directories with a-rx mode.

context: reimplement memfilectx.cmp() If I added a sanity check to basefilectx, test-context.py exploded. This patch copies the naive implementation from overlayworkingfilectx.

rust-cpython: build and support for Python3 Defined Cargo features for Python3, making them overall simpler to use, hooked them in build and made mercurial.rustext importable. This is tested with Python 3.6.7. Differential Revision: https://phab.mercurial-scm.org/D5446

py3: convert popen() command arguments in hgclient to str on Windows This fixes test-commandserver.t and test-keyword.t, which was previously complaining TypeError("a bytes-like object is required, not 'str'")

py3: byteify tests/pullext.py # skip-blame for b'' prepending

py3: byteify the fakeversion extension in test-http-bad-server.t # skip-blame for b'' prefixing

py3: spawn all python instances with legacy stdio enabled on Windows This fixes 6 more tests. See ef7119cd4965.

setup: avoid attempting to invoke the system-wide hg.exe on Windows On Windows, the executable in the current directory gets priority over anything in $PATH (both for cmd.exe and MSYS). That means, the former code was launching the local hg.exe instead of the system-wide one, if it was previously built. If that failed, it then fell back to the local hg code, but run through python.exe. I'm not sure what it is about ef7119cd4965, but that started throwing up a messagebox that python37.dll couldn't be loaded. (And indeed, python37 is not in $PATH by default.) Invoking the local hg via the current python avoids that.

delta: ignore base whose chains already don't match expectations If we know the existing chain does not match our criteria, there is no point to build a delta to append. This is especially useful when dealing with a full text much smaller than its parent. In that case, the parent chain is probably already too large. example affected manifest write before: 1.421005s after: 0.815520s (-42%)

delta: exclude base candidate much smaller than the target If a revision's full text is that much bigger than a base candidate full text, we no longer consider that candidate. This solves a pathological case we encountered on a very specify repository. It contains a long series of changesets with a very small manifest (one file) co-existing with others changesets using a very large manifest. Without this filtering, we ended up considering a large number of tiny full snapshots as a potential base. It resulted in very large delta (the size of the full text) and mercurial spending 99% of its time compressing these deltas. The timing of a commit moved from about 400s to about 10s (still slow, but not ridiculously slow).

perfrevflogwrite: clear revlog cache between each write We want to measure write time from a cold cache (similar to commit). So we need to clear the cache to prevent computation from rev N-1 to interfere with rev N.

py3: enable legacy stdio mode in exewrapper This drops the test failure count from 166 to 117. The failures were typically in the form of `hg serve -d` spawning but crashing immediately, leaving clients with "bad http status" lines, connection refusals, and so forth. The underlying message on the server side was either "OSError: [WinError 6] The handle is invalid" or "OSError: [WinError 1] Incorrect function". Additionally, no output was rendered if the pager was activated. Thanks to Yuya for diagnosing the problem. The failure count drops to 107 when PYTHONLEGACYWINDOWSSTDIO=1 is defined in the environment. These failures seem to revolve around the dummyssh server process, and dumbhttp.py. So I'll probably add that to the test runner. One small regression here (only in py3) is that if hg.exe is already built, a messagebox appears when building it again saying that python37.dll can't be loaded. Python3 isn't in PATH by default, and setup.py tries running bare `hg` first. But MSYS prepends '.' to PATH, so it runs the local hg, but can't find the library. It falls back to the python used to invoke setup.py, so ultimately it works. I'm not sure if it's better to strip '.' from PATH or just skip right to `sys.executable hg` on Windows. Also, something seems to be wrong with run-tests._usecorrectpython(). I accidentially left off the 'PYTHON="py -3"' when building (thus making py2 stuff), and yet managed to invoke run-tests.py with "py -3". (And that only had 67 failures.)

run-tests: alias hg to hg.exe on Windows To enable legacy stdio mode on Windows, hg.exe needs to be updated. But before that, we actually have to use it when running the tests. I *think* what was happening before was when MSYS invoked `hg`, it looked at the shbang line and ran python.exe found there. The test harness must be updating $PATH to include the python used to launch it, and therefore it ran py3. As a side note, this also fixed `py -3 run-tests.py` (without --local), which complained about the space in the shbang line before this. This should also help in WSL, because the explicit '.exe' is needed to invoke a Windows app instead of invoking the Linux app. I have no idea why this change capitalizes Lib in the tests, as it was previously lowercase for both py2 and py3.

py3: conditionalize the python version in test-install.t

tests: followup on network related errors on Debian 9 First, update test-clonebundles.t which was failing since the introduction of the `$EADDRNOTAVAIL$` common pattern. Also update two tests from 8695fbe17f7c with a more compact condition line. Differential Revision: https://phab.mercurial-scm.org/D5445

py3: unimplement RevlogError.__str__() On Python 2, str(exc) would crash if __str__() returned a unicode containing non-ASCII characters.

py3: use forcebytestr() to stringify hook exception This fixes test-narrow-exchange.t.

color: fix a documentation typo

py3: quote $PYTHON in test-patchbomb.t for Windows I couldn't get the quoting right in the environment variable, so now it's a function.

rust-cpython: testing the bindings from Python This is easier and more convincing than doing the same tests from a Rust tests module. Differential Revision: https://phab.mercurial-scm.org/D5437

rust-cpython: build via HGWITHRUSTEXT=cpython The existing behaviour, building the direct ffi bindings if HGIWTHRUSTEXT is just set is unchanged, but if HGWITHRUSTEXT is cpython, then the cpython bindings (aka mercurial/rustext.so) are built. Differential Revision: https://phab.mercurial-scm.org/D5436

rust: better treatment of cargo/rustc errors Differential Revision: https://phab.mercurial-scm.org/D5435

rust-cpython: start cpython crate bindings This changeset introduces the hg-cpython crate, that compiles as a shared library holding a whole Python package (mercurial.rustext), with only the empty 'ancestor' submodule for now. Such bindings will be easier and safer to develop and maintain that those of `hg-direct-ffi`. They don't involve C code, only unsafe Rust that's mostly isolated within the cpython crate. The long-term goal would be to import the provided modules, such as rustext.ancestor with mercurial.policy.importmod, same as we already do with cext modules. Differential Revision: https://phab.mercurial-scm.org/D5434

rust-cpython: exclude hgcli from workspace hgcli uses a specific rust-cpython commit by indygreg, of which a PR has been derived which is not merged nor released yet. But we can't use several versions of the sys-python2.7 crate in a single workspace: it makes for a build error. Since hgcli does not at the time being need anything from hg-core, whereas the upcoming hg-cpython will. So for now we're moving hgcli aside, hoping we could base all of them on the same version of rust-cpython again in the future. Differential Revision: https://phab.mercurial-scm.org/D5433

sparse-revlog: protect C code against delta chain including nullrev For unclear reasons, some repositories include nullrev (-1). Re-computing delta for such repo remove nullrev from all chain, so some older versions have been creating them. This currently raise an IndexError with the new C code doing chain slicing as it expect all item to be positive. Both python and C code for reading delta chain preserve nullrev, and the Python code for chain slicing handle the case fine. So we take the safe route and make the new C code works fine in that case.

sparse-revlog: handle nullrev in index_get_length The more generic index_get method handle nullrev fine, we apply the same logic here.

sparse-revlog: handle nullrev in index_get_start The more generic index_get method handle nullrev fine, we apply the same logic here.

revlog: introduce a constant for nullrev in `revlog.c` The value is important enough to be explicitly tracked.

rust: translation of missingancestors This is as direct as possible a translation of the ancestor.missingancestors Python class in pure Rust. The goal for this changeset is to make it easy to compare with the Python version. We also add to Python tests the cases that helped us develop and debug this implementation. Some possible optimizations are marked along the way as TODO comments Differential Revision: https://phab.mercurial-scm.org/D5416

contrib: provide a small script that draw performance plot We have been using this script to look into the result of various runs of the `hg perfrevlogwrite` command. It seems useful enough to be shared more widely.

delta: filter nullrev out first When picking a potential candidate, we filter them on various criteria. The "different from nullrev" criteria is very fast to compute and we should process it first.

perf: report more of the higher range in perfrevlogwrite Since the delta chain length is limited to 1000 revisions, we get a new snapshot about every 1000 revisions. If we assume that the snapshot will be most of the slowest revision, the current display (99% and max) are not very precise in their area. We now include more information about this space in the default report.

help: present boolean arguments as "--[no-]foo" This should make it much more discoverable (we document it in `hg help flags`, but most users don't think to look there). Note that flags that default to None (and not False) will not get this new presentation. We can change the defaults to False later for flags where it makes sense (probably almost all boolean flags). Differential Revision: https://phab.mercurial-scm.org/D5432

help: show "default: off" for boolean flags that default to off Differential Revision: https://phab.mercurial-scm.org/D5431

help: use "default: on" instead of "default: True" "True" feels like a Python thing and not something that users should see. Differential Revision: https://phab.mercurial-scm.org/D5430

check-commit: disallow capitalization only right after topic It should be okay to write "topic: change 'default: True' to 'default: on'". Differential Revision: https://phab.mercurial-scm.org/D5429

absorb: don't prompt to apply changes when there are none to apply Differential Revision: https://phab.mercurial-scm.org/D5428

py3: add a b'' prefix in tests/test-remotefilelog-pull-noshallow.t This makes the test pass on Python 3. # skip-blame because just b'' prefix. Differential Revision: https://phab.mercurial-scm.org/D5427

error: implement __str__ on RevlogError to fix some output defects on Py3 We open-code encoding.unimethod here to avoid cycles, and do a local import of encoding when someone str()s a RevlogError. It's not my favorite solution, but it gets the job done. Differential Revision: https://phab.mercurial-scm.org/D5426

perf: add two more missing b prefixes for Python 3 # skip-blame just two more b prefixes Differential Revision: https://phab.mercurial-scm.org/D5425

match: fix doctest to use bytes instead of str Fixes doctests on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5423

py3: use pycompat.bytechr() instead of chr() in test-fastannotate-revmap.py This makes the test pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5422

py3: add a missing b'' prefix in contrib/perf.py # skip-blame because just b'' prefixes This fixes test-contrib-perf.t on Python 3 which started failing. Differential Revision: https://phab.mercurial-scm.org/D5421

py3: use '%d' for integers instead of '%s' This should fix test-rebase-inmemory.t which started failing on Python 3 after recent changes. Differential Revision: https://phab.mercurial-scm.org/D5420

py3: whitelist another passing test caught by buildbot Differential Revision: https://phab.mercurial-scm.org/D5419

py3: fix bad escapes of sub() replacement pattern in test-template-basic.t Python 3.7 starts complaining about it. We have to double the backslash or '\x5c' to get around.

perfdiscovery: benching findcommonheads() This works between the local repo and any peer given by its path, and should be useful for further work on discovery Differential Revision: https://phab.mercurial-scm.org/D5418

windows: ensure pure posixfile fd doesn't escape by entering context manager There are tests in test-revlog-mmapindex.t and test-rebase-mq-skip.t that are fixed by this, but we usually don't use --pure on Windows. For whatever reason, the remaining --pure failures are various errors like $ENOTDIR$ and "Access is denied" have a trailing '.'.

vfs: ensure closewrapbase fh doesn't escape by entering context manager I'm not sure if there's a problem in practice here, as there's no test failure either way. The __exit__() and close() methods raise an exception, so maybe __exit__() and close() are being called directly on the underlying handle when delayclosedfile is used on a context manager? I doubt that was intended.

windows: ensure mixedfilemodewrapper fd doesn't escape by entering context mgr Otherwise it seems that the special read and write handling would be bypassed.

py3: ensure the proxied Windows fd doesn't escape by entering context manager The purpose of the proxy class is to provide the `name` attribute which contains the file path. But in tests that used a context manager, it still blew up complaining that 'int' doesn't have a 'startswith' function.

merge with stable

templatefuncs: specialize "no match" value of search() to allow % operation If Python had Maybe or Option, the type of the search() result would be Option<Mapping>, which can be considered as a 0/1 container of a Mapping. So it makes sense that {search(r'no match pattern', x) % "whatever"} is mapped to an empty string.

templatefuncs: add regexp search() function that extracts substring This can be used to extract an issue number from a commit message, for example: {search(r'\(issue([0-9]*)\)', desc) % '{1}'}

rust: changed Graph.parents to return [Revision; 2] This will allow for simple iteration on parent revisions, such as: for parent in graph.parents(rev)?.iter().cloned() This seems to be a zero overhead abstraction once built in release mode. Differential Revision: https://phab.mercurial-scm.org/D5415

rust: improved docstring In the previous wording, rustfmt wanted to cut at the == which is not very readable in my taste. Differential Revision: https://phab.mercurial-scm.org/D5414

revset: move subscript relation functions to its own dict This will help adding more relation functions in extensions. We skip short names (that consist of one letter) while raising UnknownIdentifier because such names cannot be suggested anyway: the similarity cutoff in dispatch._getsimilar() is currently 0.6.

py3: teach run-tests.py to handle exe with spaces when --local isn't specified This was the reason that no amount of quoting worked in test-hghave.t. `os.popen()` needed to be swapped out because while the added quoting around line 3124 worked on py3, it failed on py2. See 38d51371792b. The problem with `os.system()` was wrongly splitting the command on the space in 'Program Files', regardless of quoting. It looks like there are a few other instances of `os.system()` in core code, so presumably those should be replaced?

rust: adapted hg-core tests for iteration over Result Now AncestorsIterator iters on Result<Revision, GraphError>

win32: close the handles associated with a spawned child process Probably not a big deal because at this point, the call is only used when spawning a daemonized server. In that case, the parent process goes away first, so it won't prevent the child from being cleaned up.

rust: remove comment about error handling of AncestorsIterator To be align with 443eb4bc41af "rust: propagate error of index_get_parents() properly." Spotted by Georges Racinet.

test: fix test-http-bad-server with current python 2.7 https://github.com/python/cpython/pull/2825 changed the exception message for empty http status line. Differential Revision: https://phab.mercurial-scm.org/D5412

perf: add perfprogress command I've noticed that progress bars can add significant overhead to tight loops. Let's add a perf command that attempts to isolate that overhead. With a default hgrc, iteration over 1M items appears to take ~3.75s on my machine. Profiling reveals ~28% of time is spent in ui.configbool() resolving the value of the progress.debug config option. Even if I set progress.disable=true, execution still takes ~2.60s, with ~59% of the time spent in ui.configbool(). Differential Revision: https://phab.mercurial-scm.org/D5407

wireprotov2: unify file revision collection and linknode derivation The old mechanism for choosing which file revisions to send in the haveparents=True case was buggy in multiple ways - the most severe of which being that file revisions were excluded when they shouldn't have been. This commit unifies the logic for deriving the filenodes that will be sent by the "filesdata" command. We now consistently read files data from manifests. The "haveparents" argument now controls whether we iterate ctx.files() or use the full manifest to derive relevant files. The logic here is still woefully lacking to fully support shallow clones. It will require an API break to fully address. This commit should at least make the server APIs emit proper data, which is strictly better than before. Differential Revision: https://phab.mercurial-scm.org/D5406

wireprotov2: send linknodes to emitfilerevisions() Previously, linknodes were calculated within emitfilerevisions() by using filectx.introrev(), which would always use the linkrev/linknode as recorded by storage. This is wrong for cases where the receiver doesn't have the changeset the linknode refers to. This commit changes the logic for linknode emission so the mapping of filenode to linknode is computed by the caller and passed into emitfilerevisions(). As part of the change, linknodes for "filesdata" in the haveparents=False case are now correct: the existing code performed a manifest walk and it was trivial to plug in the correct linknode. However, behavior for the haveparents=True case is still wrong because it relies on filtering linkrevs against the outgoing set in order to determine what to send. This will be fixed in a subsequent commit. The change test test-wireproto-exchangev2-shallow.t is a bit wonky. The test repo has 6 revisions. The changed test is performing a shallow clone with depth=1. So, only file data for revision 5 is present locally. So, the new behavior of associating the linknode with revision 5 for every file revision seems correct. Of course, when backfilling old revisions, we'll want to update the linknode. But this problem requires wire protocol support and we'll cross that bridge later. Differential Revision: https://phab.mercurial-scm.org/D5405

tests: add tests for server-side linknode adjustment with wireprotov2 The current implementation of linknode serving in wireprotov2 simply serves up the linkrev/linknode as stored: it doesn't attempt to adjust the linknode to what the receiver is aware of. This can result in the client seeing a linknode referencing a changeset that is unknown to it. This commit adds test coverage of that scenario. The tests in test-wireproto-command-filesdata.t demonstrate two failures. First, the linknode refers to a changeset not in the available set. Second, the server doesn't send a file revision that it should have (because of linkrev filtering). The test in test-wireproto-exchange.t demonstrates that the lack of a file revision results in a corrupted repository on the client. Differential Revision: https://phab.mercurial-scm.org/D5404

tests: fix empty commit in test This was almost certainly a bug. Differential Revision: https://phab.mercurial-scm.org/D5403

tests: add sparserevlog requirement to test-sqlitestore.t Looks like this test was missed when enabling sparse revlogs by default. Differential Revision: https://phab.mercurial-scm.org/D5402

tests: remove all transitional configuration Now that sparse-revlog is enabled by default, we no longer needs it. Differential Revision: https://phab.mercurial-scm.org/D5346

sparse-revlog: enabled by default The feature provides large benefits. It now seems mature enough to be enabled by default. * It solves catastrophic issues regarding delta storage in revlog, * It allows for shorter delta chain in all repositories, improving performances. Running benchmark of a wide range of operation did not reveal problematic impact. Performance gains are observed where expected. The format is supported by Mercurial version 4.7. So it seems safe to enable it by default now. Here is a reminder of key numbers regarding this delta strategy effect on repository size and performance. Effect on Size: =============== For repositories with a lot of branches, sparse-revlog significantly improve size, fixing limitation associated with the span of a delta chain. In addition, sparse-revlog, deal well with limitations of the delta chain length. For large repositories, this allows for a stiff reduction of the delta chain without a problematic impact on the repository size. This delta chain length improvement helps all repositories, not just the ones with many branches. As a reminder, here are the default chain limits for each "format": * no-sparse: none * sparse: 1000 Mercurial --------- Manifest Size: limit | none | 1000 ------------|-------------|------------ no-sparse | 6 143 044 | 6 269 496 sparse | 5 798 796 | 5 827 025 Manifest Chain length data limit || none || 1000 value || average | max || average | max ------------||---------|---------||---------|--------- no-sparse || 429 | 1 397 || 397 | 1 000 sparse || 326 | 1 290 || 313 | 1 000 Full Store Size limit | none | 1000 ------------|-------------|------------ no-sparse | 46 944 775 | 47 166 129 sparse | 46 622 445 | 46 723 774 pypy ---- Manifest Size: limit | none | 1000 ------------|-------------|------------ no-sparse | 52 941 760 | 56 200 970 sparse | 26 348 229 | 27 384 133 Manifest Chain length data limit || none || 1000 value || average | max || average | max ------------||---------|---------||---------|--------- no-sparse || 769 | 3 889 || 390 | 1 000 sparse || 1 223 | 3 846 || 495 | 1 000 Full Store Size limit | none | 1000 ------------|-------------|------------ no-sparse | 336 050 203 | 339 309 413 sparse | 338 673 985 | 339 709 889 Mozilla ------- Manifest Size: limit | none | 1000 ------------|----------------|--------------- no-sparse | 215 096 339 | 1 708 853 525 sparse | 188 947 271 | 278 894 170 Manifest Chain length data limit || none || 1000 value || average | max || average | max ------------||---------|---------||---------|-------- no-sparse || 20 454 | 59 562 || 491 | 1 000 sparse || 23 509 | 69 891 || 489 | 1 000 Full Store Size limit | none | 1000 ------------|----------------|--------------- no-sparse | 2 377 578 715 | 3 876 258 798 sparse | 2 441 677 137 | 2 535 997 381 Netbeans -------- Manifest Size: limit | none | 1000 ------------|----------------|--------------- no-sparse | 130 088 982 | 741 590 565 sparse | 118 836 887 | 159 161 207 Manifest Chain length data limit || none || 1000 value || average | max || average | max ------------||---------|---------||---------|--------- no-sparse || 19 321 | 61 397 || 510 | 1 000 sparse || 21 240 | 61 583 || 503 | 1 000 Full Store Size limit | none | 1000 ------------|----------------|--------------- no-sparse | 1 160 013 008 | 1 771 514 591 sparse | 1 164 959 988 | 1 205 284 308 Private repo #1 --------------- Manifest Size: limit | none | 1000 ------------|-----------------|--------------- no-sparse | 33 725 285 081 | 33 724 834 190 sparse | 350 542 420 | 423 470 579 Manifest Chain length data limit || none || 1000 value || average | max || average | max ------------||---------|---------||---------|--------- no-sparse || 282 | 8 885 || 113 | 1 000 snapshot || 3 655 | 8 951 || 530 | 1 000 Full Store Size limit | none | 1000 ------------|----------------|--------------- no-sparse | 41 544 149 652 | 41 543 698 761 sparse | 8 448 037 300 | 8 520 965 459 Effect on speed: ================ Performances are strongly impacted by the delta chain length. Longer chain results in slower revision restoration. For this reason, the 1000 chain limit introduced by sparse-revlog helps repository with previously large chains a lot. In our corpus, this means `netbeans` and `mozilla-central` who suffered from unreasonable manifest delta chain length. Another way sparse revlog helps, is by producing better delta's. For repositories with many branches, the pathological patterns that resulted in many sub-optimal deltas are gone. Smaller delta help with operations where deltas are directly relevant, like bundle. However, the sparse-revlog logic introduces some extra processing and a more throughout testing of possible delta candidates. Adding an extra cost in some cases. This cost is usually counterbalanced by the other performance gain. However, for smaller repositories not affected by delta chain length issues or branching related issues, this might make things a bit slower. However, these are also repository where revlog performance is dwarfed by other costs. Below are the summary of some timing from the performance test suite running at `http://perf.octobus.net/` for a handful of key commands and operation. It is important to keep in mind that most of this command works on the tip part of the repository. The non-sparse and sparse version produce different delta chains and the tip revision can end up at an arbitrary point of these chains. This will impact some performance number listed in this summary. For the record: here is the delta chain length for the tip revision of manifest log in the benchmarked repository: | no-sparse | sparse | mercurial | 94 | 904 | pypy | 23 | 673 | netbeans | 4158 | 258 | mozilla | 63263 | 781 | As you can see, the chain length for mercurial and pypy turn out to be significantly longer. The netbeans and mozilla one get shorter because these repositories benefit from the maximum chain length. Timing for `hg commit`: ----------------------- The time taken by `hg commit` does not varies significantly, no drawback for using sparse here. | no-sparse | sparse | mercurial | 68.1ms | 66.7ms | pypy | 95.0ms | 94.1ms | netbeans | 614.0ms | 611.0ms | mozilla | 1340.0ms | 1.320.0ms | Check the final section for statistics on a wider array of write. Timing for bundling 10 000 changesets ------------------------------------- The repository that benefits from better delta see a good performance boost. The other ones are not significantly affected. | no-sparse | sparse | mercurial | 3.1s | 3.0s | pypy | 25.1s | 7.5s | netbeans | 24.2s | 17.0s | mozilla | 23.7s | 25.0s | Timing for unbundling 1 000 changesets -------------------------------------- Mercurial and mozilla are unaffected. The pypy repository benefit well from the better delta. However, the netbeans repository takes a visible hit. Digging that difference reveals that it comes from the sparse-revlog bundle having to deal with a snapshot that was re-encoded in the bundle. The slow path for adding new a revision had to be triggered for it, slowing things down. The Sparse versions do not have such snapshot to handle similar cases in the tested configuration. | no-sparse | sparse | mercurial | 519ms | 502ms | pypy | 1.270ms | 886ms | netbeans | 1.370ms | 2.250ms | mozilla | 3.230ms | 3.210ms | Netbeans benefits from the better deltas in other dimensions too. For example, the produced bundle is significantly smaller: * netbeans-no-sparse.hg: 2.3MB * netbeans-sparse.hg: 1.9MB Timing to restore the tip most manifest entry: ---------------------------------------------- Nothing surprising here. The timing for mercurial and pypy are within a small range where they won't affect performance much. In our tested case, they are slower as they use a longer chain. Timing for netbeans and mozilla improves a lot. Removing a significant amount of time. | no-sparse | sparse | mercurial | 1.09ms | 3.15ms | pypy | 4.11ms | 10.70ms | netbeans | 239.00ms | 112.00ms | mozilla | 688.00ms | 198.00ms | Reading 100 revision in descending order: ----------------------------------------- We see the same kind of effect when reading the last 100 revisions. Large boost for netbeans and mozilla, as they use much smaller delta chain. Mercurial and pypy longer chain means slower reads, but nothing gets out of control. | no-sparse | sparse | mercurial | 0.089s | 0.268s | pypy | 0.259s | 0.698s | netbeans | 125.000s | 20.600s | mozilla | 23.000s | 11.400s | Writing from full text: statistic for the last 30K revisions ------------------------------------------------------------ This benchmark adds revisions to revlog from their full text. This is similar to the work done during a commit, but for a large amount of revisions so that we get a more relevant view. We see better overall performances with sparse-revlog. The very worst case is usually slower with sparse-revlog, but does not gets out of control. For the vast majorities of the other writes, sparse-revlog is significantly faster for larger repositories. This is reflected in the accumulated rewrite time for netbeans and mozilla. The notable exception is the pypy repository where things get slower. The extra processing is not balanced by shorter delta chain. However, this is to be seen as a blocking issue. First, the overall time spend dealing with revlog for the repository pypy size is small compared to the other costs, so we get slower on operations that matter less than for other larger repository. Second, we still get nice size benefit from using sparse-revlog, smaller repo size brings other usability and speed benefit (eg: bundle size). max time | no-sparse | sparse | mercurial | 0.010143s | 0.045280s | pypy | 0.034924s | 0.243288s | netbeans | 0.605371s | 2.130876s | mozilla | 1.478342s | 3.424541s | 99% time | no-sparse | sparse | mercurial | 0.003774s | 0.003758s | pypy | 0.017387s | 0.025310s | netbeans | 0.576913s | 0.271195s | mozilla | 1.478342s | 0.449661s | 95% time | no-sparse | sparse | mercurial | 0.002069s | 0.002120s | pypy | 0.010141s | 0.014797s | netbeans | 0.540202s | 0.258644s | mozilla | 0.654830s | 0.243440s | full time | no-sparse | sparse | mercurial | 14.15s | 14.87s | pypy | 90.50s | 137.12s | netbeans | 6401.06s | 3411.14s | mozilla | 3086.89s | 1991.97s | Differential Revision: https://phab.mercurial-scm.org/D5345

test: preemptively disable sparse-revlog for some of test-upgrade-repo Some part of this test assumes sparse-revlog is off. We enforce it explicitly. Differential Revision: https://phab.mercurial-scm.org/D5344

test: enable sparse-revlog for test-wireproto-content-redirects.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5343

test: enable sparse-revlog for test-wireproto-command-capabilities.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5342

test: enable sparse-revlog for test-wireproto-caching.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5341

test: enable sparse-revlog for test-treemanifest.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5340

test: enable sparse-revlog for test-stream-bundle-v2.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5339

test: enable sparse-revlog for test-ssh.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5338

test: enable sparse-revlog for test-ssh-proto-unbundle.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5337

test: enable sparse-revlog for test-ssh-proto.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5336

test: enable sparse-revlog for test-ssh-bundle1.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5335

test: enable sparse-revlog for test-sparse-requirement.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5334

test: enable sparse-revlog for test-revlog-v2.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5333

test: enable sparse-revlog for test-repo-compengines.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5332

test: enable sparse-revlog for test-remotefilelog-log.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5331

test: enable sparse-revlog for test-remotefilelog-clone-tree.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5330

test: enable sparse-revlog for test-remotefilelog-clone.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5329

test: enable sparse-revlog for test-phases.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5328

test: enable sparse-revlog for test-narrow-clone.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5327

test: enable sparse-revlog for test-narrow-clone-stream.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5326

test: enable sparse-revlog for test-narrow-clone-no-ellipsis.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5325

test: enable sparse-revlog for test-lfs-serve.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5324

test: enable sparse-revlog for test-lfs-largefiles.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5323

test: enable sparse-revlog for test-lfconvert.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5322

test: enable sparse-revlog for test-init.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5321

test: enable sparse-revlog for test-http-protocol.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5320

test: enable sparse-revlog for test-hgweb-commands.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5319

test: enable sparse-revlog for test-debugcommands.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5318

test: enable sparse-revlog for test-clone-uncompressed.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5317

test: enable sparse-revlog for test-clonebundles.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5316

test: enable sparse-revlog for test-bundle.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5315

test: make sure sparse-revlog does not interfer with test-parseindex.t That test carefully craft revlog to reproduce issues. We do not want new revlog behavior to interfere with that. Differential Revision: https://phab.mercurial-scm.org/D5314

test: explicitly disable sparse-revlog in test-http-bad-server.t The `test-http-bad-server.t` test is very fragile with hard-coded number all around. Adding any new capability have massive consequences on it that are long and tiresome to handle. Since the test check more generic protocol level behavior, it does not need to follow all new capabilities. Disabling the capabilities is a much simpler way to avoid the issue. We make sure `sparse-revlog` will not impact this test once turned on by default. Differential Revision: https://phab.mercurial-scm.org/D5313

test: explicitly disable sparse-revlog in test-generaldelta.t That test is about testing generaldelta without further improvement. We ensure it will stay that way even if sparse-revlog is turned on by default. The sparse-revlog format is tested on his own in a different test file. Differential Revision: https://phab.mercurial-scm.org/D5312

sparse-revlog: disable sparse-revlog if config disable general-delta Sparse-revlog requires general-delta in order to work. If general-delta is explicitly disabled through configuration, we should honor that and also disabled sparse-revlog. This change will be more relevant when sparse-revlog is enabled by default. Differential Revision: https://phab.mercurial-scm.org/D5311

test: fix config typo in test-upgrade-repo.t Differential Revision: https://phab.mercurial-scm.org/D5310

tests: use $RUNTESTDIR instead of $TESTDIR in narrow-library.sh This is done so that third party extensions can use narrow-library.sh in tests. Differential Revision: https://phab.mercurial-scm.org/D5400

tests: update network related errors for Debian 9 We have a CI job that runs the Mercurial tests in parallel. Some of the network related failures seems to be different on the environment. Oddly, those failures happens only when running the tests in parallel, not when running the test file only. I have no idea how to get the windows formatted message for the error, if someone could give me an hand, I will update this changeset with the value. Differential Revision: https://phab.mercurial-scm.org/D5401

util: implement pop() on lrucachedict This moves __delitem__() to pop() as the requirement is pretty much the same, and reimplement __delitem__() by using pop().

util: add method to peek item in lrucachedict I want a function that doesn't unnecessarily update the internal state of the cache dict after fork().

commandserver: loop over selector events An IPC socket will be waited by the same selector.