help: present boolean arguments as "--[no-]foo" This should make it much more discoverable (we document it in `hg help flags`, but most users don't think to look there). Note that flags that default to None (and not False) will not get this new presentation. We can change the defaults to False later for flags where it makes sense (probably almost all boolean flags). Differential Revision: https://phab.mercurial-scm.org/D5432

help: show "default: off" for boolean flags that default to off Differential Revision: https://phab.mercurial-scm.org/D5431

help: use "default: on" instead of "default: True" "True" feels like a Python thing and not something that users should see. Differential Revision: https://phab.mercurial-scm.org/D5430

check-commit: disallow capitalization only right after topic It should be okay to write "topic: change 'default: True' to 'default: on'". Differential Revision: https://phab.mercurial-scm.org/D5429

absorb: don't prompt to apply changes when there are none to apply Differential Revision: https://phab.mercurial-scm.org/D5428

py3: add a b'' prefix in tests/test-remotefilelog-pull-noshallow.t This makes the test pass on Python 3. # skip-blame because just b'' prefix. Differential Revision: https://phab.mercurial-scm.org/D5427

error: implement __str__ on RevlogError to fix some output defects on Py3 We open-code encoding.unimethod here to avoid cycles, and do a local import of encoding when someone str()s a RevlogError. It's not my favorite solution, but it gets the job done. Differential Revision: https://phab.mercurial-scm.org/D5426

perf: add two more missing b prefixes for Python 3 # skip-blame just two more b prefixes Differential Revision: https://phab.mercurial-scm.org/D5425

match: fix doctest to use bytes instead of str Fixes doctests on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5423

py3: use pycompat.bytechr() instead of chr() in test-fastannotate-revmap.py This makes the test pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5422

py3: add a missing b'' prefix in contrib/perf.py # skip-blame because just b'' prefixes This fixes test-contrib-perf.t on Python 3 which started failing. Differential Revision: https://phab.mercurial-scm.org/D5421

py3: use '%d' for integers instead of '%s' This should fix test-rebase-inmemory.t which started failing on Python 3 after recent changes. Differential Revision: https://phab.mercurial-scm.org/D5420

py3: whitelist another passing test caught by buildbot Differential Revision: https://phab.mercurial-scm.org/D5419

py3: fix bad escapes of sub() replacement pattern in test-template-basic.t Python 3.7 starts complaining about it. We have to double the backslash or '\x5c' to get around.

perfdiscovery: benching findcommonheads() This works between the local repo and any peer given by its path, and should be useful for further work on discovery Differential Revision: https://phab.mercurial-scm.org/D5418

windows: ensure pure posixfile fd doesn't escape by entering context manager There are tests in test-revlog-mmapindex.t and test-rebase-mq-skip.t that are fixed by this, but we usually don't use --pure on Windows. For whatever reason, the remaining --pure failures are various errors like $ENOTDIR$ and "Access is denied" have a trailing '.'.

vfs: ensure closewrapbase fh doesn't escape by entering context manager I'm not sure if there's a problem in practice here, as there's no test failure either way. The __exit__() and close() methods raise an exception, so maybe __exit__() and close() are being called directly on the underlying handle when delayclosedfile is used on a context manager? I doubt that was intended.

windows: ensure mixedfilemodewrapper fd doesn't escape by entering context mgr Otherwise it seems that the special read and write handling would be bypassed.

py3: ensure the proxied Windows fd doesn't escape by entering context manager The purpose of the proxy class is to provide the `name` attribute which contains the file path. But in tests that used a context manager, it still blew up complaining that 'int' doesn't have a 'startswith' function.

merge with stable

templatefuncs: specialize "no match" value of search() to allow % operation If Python had Maybe or Option, the type of the search() result would be Option<Mapping>, which can be considered as a 0/1 container of a Mapping. So it makes sense that {search(r'no match pattern', x) % "whatever"} is mapped to an empty string.

templatefuncs: add regexp search() function that extracts substring This can be used to extract an issue number from a commit message, for example: {search(r'\(issue([0-9]*)\)', desc) % '{1}'}

rust: changed Graph.parents to return [Revision; 2] This will allow for simple iteration on parent revisions, such as: for parent in graph.parents(rev)?.iter().cloned() This seems to be a zero overhead abstraction once built in release mode. Differential Revision: https://phab.mercurial-scm.org/D5415

rust: improved docstring In the previous wording, rustfmt wanted to cut at the == which is not very readable in my taste. Differential Revision: https://phab.mercurial-scm.org/D5414

revset: move subscript relation functions to its own dict This will help adding more relation functions in extensions. We skip short names (that consist of one letter) while raising UnknownIdentifier because such names cannot be suggested anyway: the similarity cutoff in dispatch._getsimilar() is currently 0.6.

py3: teach run-tests.py to handle exe with spaces when --local isn't specified This was the reason that no amount of quoting worked in test-hghave.t. `os.popen()` needed to be swapped out because while the added quoting around line 3124 worked on py3, it failed on py2. See 38d51371792b. The problem with `os.system()` was wrongly splitting the command on the space in 'Program Files', regardless of quoting. It looks like there are a few other instances of `os.system()` in core code, so presumably those should be replaced?

rust: adapted hg-core tests for iteration over Result Now AncestorsIterator iters on Result<Revision, GraphError>

win32: close the handles associated with a spawned child process Probably not a big deal because at this point, the call is only used when spawning a daemonized server. In that case, the parent process goes away first, so it won't prevent the child from being cleaned up.

rust: remove comment about error handling of AncestorsIterator To be align with 443eb4bc41af "rust: propagate error of index_get_parents() properly." Spotted by Georges Racinet.

test: fix test-http-bad-server with current python 2.7 https://github.com/python/cpython/pull/2825 changed the exception message for empty http status line. Differential Revision: https://phab.mercurial-scm.org/D5412

perf: add perfprogress command I've noticed that progress bars can add significant overhead to tight loops. Let's add a perf command that attempts to isolate that overhead. With a default hgrc, iteration over 1M items appears to take ~3.75s on my machine. Profiling reveals ~28% of time is spent in ui.configbool() resolving the value of the progress.debug config option. Even if I set progress.disable=true, execution still takes ~2.60s, with ~59% of the time spent in ui.configbool(). Differential Revision: https://phab.mercurial-scm.org/D5407

wireprotov2: unify file revision collection and linknode derivation The old mechanism for choosing which file revisions to send in the haveparents=True case was buggy in multiple ways - the most severe of which being that file revisions were excluded when they shouldn't have been. This commit unifies the logic for deriving the filenodes that will be sent by the "filesdata" command. We now consistently read files data from manifests. The "haveparents" argument now controls whether we iterate ctx.files() or use the full manifest to derive relevant files. The logic here is still woefully lacking to fully support shallow clones. It will require an API break to fully address. This commit should at least make the server APIs emit proper data, which is strictly better than before. Differential Revision: https://phab.mercurial-scm.org/D5406

wireprotov2: send linknodes to emitfilerevisions() Previously, linknodes were calculated within emitfilerevisions() by using filectx.introrev(), which would always use the linkrev/linknode as recorded by storage. This is wrong for cases where the receiver doesn't have the changeset the linknode refers to. This commit changes the logic for linknode emission so the mapping of filenode to linknode is computed by the caller and passed into emitfilerevisions(). As part of the change, linknodes for "filesdata" in the haveparents=False case are now correct: the existing code performed a manifest walk and it was trivial to plug in the correct linknode. However, behavior for the haveparents=True case is still wrong because it relies on filtering linkrevs against the outgoing set in order to determine what to send. This will be fixed in a subsequent commit. The change test test-wireproto-exchangev2-shallow.t is a bit wonky. The test repo has 6 revisions. The changed test is performing a shallow clone with depth=1. So, only file data for revision 5 is present locally. So, the new behavior of associating the linknode with revision 5 for every file revision seems correct. Of course, when backfilling old revisions, we'll want to update the linknode. But this problem requires wire protocol support and we'll cross that bridge later. Differential Revision: https://phab.mercurial-scm.org/D5405

tests: add tests for server-side linknode adjustment with wireprotov2 The current implementation of linknode serving in wireprotov2 simply serves up the linkrev/linknode as stored: it doesn't attempt to adjust the linknode to what the receiver is aware of. This can result in the client seeing a linknode referencing a changeset that is unknown to it. This commit adds test coverage of that scenario. The tests in test-wireproto-command-filesdata.t demonstrate two failures. First, the linknode refers to a changeset not in the available set. Second, the server doesn't send a file revision that it should have (because of linkrev filtering). The test in test-wireproto-exchange.t demonstrates that the lack of a file revision results in a corrupted repository on the client. Differential Revision: https://phab.mercurial-scm.org/D5404

tests: fix empty commit in test This was almost certainly a bug. Differential Revision: https://phab.mercurial-scm.org/D5403

tests: add sparserevlog requirement to test-sqlitestore.t Looks like this test was missed when enabling sparse revlogs by default. Differential Revision: https://phab.mercurial-scm.org/D5402

tests: remove all transitional configuration Now that sparse-revlog is enabled by default, we no longer needs it. Differential Revision: https://phab.mercurial-scm.org/D5346

sparse-revlog: enabled by default The feature provides large benefits. It now seems mature enough to be enabled by default. * It solves catastrophic issues regarding delta storage in revlog, * It allows for shorter delta chain in all repositories, improving performances. Running benchmark of a wide range of operation did not reveal problematic impact. Performance gains are observed where expected. The format is supported by Mercurial version 4.7. So it seems safe to enable it by default now. Here is a reminder of key numbers regarding this delta strategy effect on repository size and performance. Effect on Size: =============== For repositories with a lot of branches, sparse-revlog significantly improve size, fixing limitation associated with the span of a delta chain. In addition, sparse-revlog, deal well with limitations of the delta chain length. For large repositories, this allows for a stiff reduction of the delta chain without a problematic impact on the repository size. This delta chain length improvement helps all repositories, not just the ones with many branches. As a reminder, here are the default chain limits for each "format": * no-sparse: none * sparse: 1000 Mercurial --------- Manifest Size: limit | none | 1000 ------------|-------------|------------ no-sparse | 6 143 044 | 6 269 496 sparse | 5 798 796 | 5 827 025 Manifest Chain length data limit || none || 1000 value || average | max || average | max ------------||---------|---------||---------|--------- no-sparse || 429 | 1 397 || 397 | 1 000 sparse || 326 | 1 290 || 313 | 1 000 Full Store Size limit | none | 1000 ------------|-------------|------------ no-sparse | 46 944 775 | 47 166 129 sparse | 46 622 445 | 46 723 774 pypy ---- Manifest Size: limit | none | 1000 ------------|-------------|------------ no-sparse | 52 941 760 | 56 200 970 sparse | 26 348 229 | 27 384 133 Manifest Chain length data limit || none || 1000 value || average | max || average | max ------------||---------|---------||---------|--------- no-sparse || 769 | 3 889 || 390 | 1 000 sparse || 1 223 | 3 846 || 495 | 1 000 Full Store Size limit | none | 1000 ------------|-------------|------------ no-sparse | 336 050 203 | 339 309 413 sparse | 338 673 985 | 339 709 889 Mozilla ------- Manifest Size: limit | none | 1000 ------------|----------------|--------------- no-sparse | 215 096 339 | 1 708 853 525 sparse | 188 947 271 | 278 894 170 Manifest Chain length data limit || none || 1000 value || average | max || average | max ------------||---------|---------||---------|-------- no-sparse || 20 454 | 59 562 || 491 | 1 000 sparse || 23 509 | 69 891 || 489 | 1 000 Full Store Size limit | none | 1000 ------------|----------------|--------------- no-sparse | 2 377 578 715 | 3 876 258 798 sparse | 2 441 677 137 | 2 535 997 381 Netbeans -------- Manifest Size: limit | none | 1000 ------------|----------------|--------------- no-sparse | 130 088 982 | 741 590 565 sparse | 118 836 887 | 159 161 207 Manifest Chain length data limit || none || 1000 value || average | max || average | max ------------||---------|---------||---------|--------- no-sparse || 19 321 | 61 397 || 510 | 1 000 sparse || 21 240 | 61 583 || 503 | 1 000 Full Store Size limit | none | 1000 ------------|----------------|--------------- no-sparse | 1 160 013 008 | 1 771 514 591 sparse | 1 164 959 988 | 1 205 284 308 Private repo #1 --------------- Manifest Size: limit | none | 1000 ------------|-----------------|--------------- no-sparse | 33 725 285 081 | 33 724 834 190 sparse | 350 542 420 | 423 470 579 Manifest Chain length data limit || none || 1000 value || average | max || average | max ------------||---------|---------||---------|--------- no-sparse || 282 | 8 885 || 113 | 1 000 snapshot || 3 655 | 8 951 || 530 | 1 000 Full Store Size limit | none | 1000 ------------|----------------|--------------- no-sparse | 41 544 149 652 | 41 543 698 761 sparse | 8 448 037 300 | 8 520 965 459 Effect on speed: ================ Performances are strongly impacted by the delta chain length. Longer chain results in slower revision restoration. For this reason, the 1000 chain limit introduced by sparse-revlog helps repository with previously large chains a lot. In our corpus, this means `netbeans` and `mozilla-central` who suffered from unreasonable manifest delta chain length. Another way sparse revlog helps, is by producing better delta's. For repositories with many branches, the pathological patterns that resulted in many sub-optimal deltas are gone. Smaller delta help with operations where deltas are directly relevant, like bundle. However, the sparse-revlog logic introduces some extra processing and a more throughout testing of possible delta candidates. Adding an extra cost in some cases. This cost is usually counterbalanced by the other performance gain. However, for smaller repositories not affected by delta chain length issues or branching related issues, this might make things a bit slower. However, these are also repository where revlog performance is dwarfed by other costs. Below are the summary of some timing from the performance test suite running at `http://perf.octobus.net/` for a handful of key commands and operation. It is important to keep in mind that most of this command works on the tip part of the repository. The non-sparse and sparse version produce different delta chains and the tip revision can end up at an arbitrary point of these chains. This will impact some performance number listed in this summary. For the record: here is the delta chain length for the tip revision of manifest log in the benchmarked repository: | no-sparse | sparse | mercurial | 94 | 904 | pypy | 23 | 673 | netbeans | 4158 | 258 | mozilla | 63263 | 781 | As you can see, the chain length for mercurial and pypy turn out to be significantly longer. The netbeans and mozilla one get shorter because these repositories benefit from the maximum chain length. Timing for `hg commit`: ----------------------- The time taken by `hg commit` does not varies significantly, no drawback for using sparse here. | no-sparse | sparse | mercurial | 68.1ms | 66.7ms | pypy | 95.0ms | 94.1ms | netbeans | 614.0ms | 611.0ms | mozilla | 1340.0ms | 1.320.0ms | Check the final section for statistics on a wider array of write. Timing for bundling 10 000 changesets ------------------------------------- The repository that benefits from better delta see a good performance boost. The other ones are not significantly affected. | no-sparse | sparse | mercurial | 3.1s | 3.0s | pypy | 25.1s | 7.5s | netbeans | 24.2s | 17.0s | mozilla | 23.7s | 25.0s | Timing for unbundling 1 000 changesets -------------------------------------- Mercurial and mozilla are unaffected. The pypy repository benefit well from the better delta. However, the netbeans repository takes a visible hit. Digging that difference reveals that it comes from the sparse-revlog bundle having to deal with a snapshot that was re-encoded in the bundle. The slow path for adding new a revision had to be triggered for it, slowing things down. The Sparse versions do not have such snapshot to handle similar cases in the tested configuration. | no-sparse | sparse | mercurial | 519ms | 502ms | pypy | 1.270ms | 886ms | netbeans | 1.370ms | 2.250ms | mozilla | 3.230ms | 3.210ms | Netbeans benefits from the better deltas in other dimensions too. For example, the produced bundle is significantly smaller: * netbeans-no-sparse.hg: 2.3MB * netbeans-sparse.hg: 1.9MB Timing to restore the tip most manifest entry: ---------------------------------------------- Nothing surprising here. The timing for mercurial and pypy are within a small range where they won't affect performance much. In our tested case, they are slower as they use a longer chain. Timing for netbeans and mozilla improves a lot. Removing a significant amount of time. | no-sparse | sparse | mercurial | 1.09ms | 3.15ms | pypy | 4.11ms | 10.70ms | netbeans | 239.00ms | 112.00ms | mozilla | 688.00ms | 198.00ms | Reading 100 revision in descending order: ----------------------------------------- We see the same kind of effect when reading the last 100 revisions. Large boost for netbeans and mozilla, as they use much smaller delta chain. Mercurial and pypy longer chain means slower reads, but nothing gets out of control. | no-sparse | sparse | mercurial | 0.089s | 0.268s | pypy | 0.259s | 0.698s | netbeans | 125.000s | 20.600s | mozilla | 23.000s | 11.400s | Writing from full text: statistic for the last 30K revisions ------------------------------------------------------------ This benchmark adds revisions to revlog from their full text. This is similar to the work done during a commit, but for a large amount of revisions so that we get a more relevant view. We see better overall performances with sparse-revlog. The very worst case is usually slower with sparse-revlog, but does not gets out of control. For the vast majorities of the other writes, sparse-revlog is significantly faster for larger repositories. This is reflected in the accumulated rewrite time for netbeans and mozilla. The notable exception is the pypy repository where things get slower. The extra processing is not balanced by shorter delta chain. However, this is to be seen as a blocking issue. First, the overall time spend dealing with revlog for the repository pypy size is small compared to the other costs, so we get slower on operations that matter less than for other larger repository. Second, we still get nice size benefit from using sparse-revlog, smaller repo size brings other usability and speed benefit (eg: bundle size). max time | no-sparse | sparse | mercurial | 0.010143s | 0.045280s | pypy | 0.034924s | 0.243288s | netbeans | 0.605371s | 2.130876s | mozilla | 1.478342s | 3.424541s | 99% time | no-sparse | sparse | mercurial | 0.003774s | 0.003758s | pypy | 0.017387s | 0.025310s | netbeans | 0.576913s | 0.271195s | mozilla | 1.478342s | 0.449661s | 95% time | no-sparse | sparse | mercurial | 0.002069s | 0.002120s | pypy | 0.010141s | 0.014797s | netbeans | 0.540202s | 0.258644s | mozilla | 0.654830s | 0.243440s | full time | no-sparse | sparse | mercurial | 14.15s | 14.87s | pypy | 90.50s | 137.12s | netbeans | 6401.06s | 3411.14s | mozilla | 3086.89s | 1991.97s | Differential Revision: https://phab.mercurial-scm.org/D5345

test: preemptively disable sparse-revlog for some of test-upgrade-repo Some part of this test assumes sparse-revlog is off. We enforce it explicitly. Differential Revision: https://phab.mercurial-scm.org/D5344

test: enable sparse-revlog for test-wireproto-content-redirects.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5343

test: enable sparse-revlog for test-wireproto-command-capabilities.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5342

test: enable sparse-revlog for test-wireproto-caching.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5341

test: enable sparse-revlog for test-treemanifest.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5340

test: enable sparse-revlog for test-stream-bundle-v2.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5339

test: enable sparse-revlog for test-ssh.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5338

test: enable sparse-revlog for test-ssh-proto-unbundle.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5337

test: enable sparse-revlog for test-ssh-proto.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5336

test: enable sparse-revlog for test-ssh-bundle1.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5335

test: enable sparse-revlog for test-sparse-requirement.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5334

test: enable sparse-revlog for test-revlog-v2.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5333

test: enable sparse-revlog for test-repo-compengines.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5332

test: enable sparse-revlog for test-remotefilelog-log.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5331

test: enable sparse-revlog for test-remotefilelog-clone-tree.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5330

test: enable sparse-revlog for test-remotefilelog-clone.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5329

test: enable sparse-revlog for test-phases.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5328

test: enable sparse-revlog for test-narrow-clone.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5327

test: enable sparse-revlog for test-narrow-clone-stream.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5326

test: enable sparse-revlog for test-narrow-clone-no-ellipsis.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5325

test: enable sparse-revlog for test-lfs-serve.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5324

test: enable sparse-revlog for test-lfs-largefiles.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5323

test: enable sparse-revlog for test-lfconvert.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5322

test: enable sparse-revlog for test-init.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5321

test: enable sparse-revlog for test-http-protocol.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5320

test: enable sparse-revlog for test-hgweb-commands.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5319

test: enable sparse-revlog for test-debugcommands.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5318

test: enable sparse-revlog for test-clone-uncompressed.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5317

test: enable sparse-revlog for test-clonebundles.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5316

test: enable sparse-revlog for test-bundle.t We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5315

test: make sure sparse-revlog does not interfer with test-parseindex.t That test carefully craft revlog to reproduce issues. We do not want new revlog behavior to interfere with that. Differential Revision: https://phab.mercurial-scm.org/D5314

test: explicitly disable sparse-revlog in test-http-bad-server.t The `test-http-bad-server.t` test is very fragile with hard-coded number all around. Adding any new capability have massive consequences on it that are long and tiresome to handle. Since the test check more generic protocol level behavior, it does not need to follow all new capabilities. Disabling the capabilities is a much simpler way to avoid the issue. We make sure `sparse-revlog` will not impact this test once turned on by default. Differential Revision: https://phab.mercurial-scm.org/D5313

test: explicitly disable sparse-revlog in test-generaldelta.t That test is about testing generaldelta without further improvement. We ensure it will stay that way even if sparse-revlog is turned on by default. The sparse-revlog format is tested on his own in a different test file. Differential Revision: https://phab.mercurial-scm.org/D5312

sparse-revlog: disable sparse-revlog if config disable general-delta Sparse-revlog requires general-delta in order to work. If general-delta is explicitly disabled through configuration, we should honor that and also disabled sparse-revlog. This change will be more relevant when sparse-revlog is enabled by default. Differential Revision: https://phab.mercurial-scm.org/D5311

test: fix config typo in test-upgrade-repo.t Differential Revision: https://phab.mercurial-scm.org/D5310

tests: use $RUNTESTDIR instead of $TESTDIR in narrow-library.sh This is done so that third party extensions can use narrow-library.sh in tests. Differential Revision: https://phab.mercurial-scm.org/D5400

tests: update network related errors for Debian 9 We have a CI job that runs the Mercurial tests in parallel. Some of the network related failures seems to be different on the environment. Oddly, those failures happens only when running the tests in parallel, not when running the test file only. I have no idea how to get the windows formatted message for the error, if someone could give me an hand, I will update this changeset with the value. Differential Revision: https://phab.mercurial-scm.org/D5401

util: implement pop() on lrucachedict This moves __delitem__() to pop() as the requirement is pretty much the same, and reimplement __delitem__() by using pop().

util: add method to peek item in lrucachedict I want a function that doesn't unnecessarily update the internal state of the cache dict after fork().

commandserver: loop over selector events An IPC socket will be waited by the same selector.

commandserver: remove redundant "if True" block

commandserver: extract handler of new socket connection This prepares for adding an IPC socket.

commandserver: pass around option to hook repo instance creation This is necessary to wrap a repo instance so the master process will be notified on repo.close().

py3: stop subscripting socket.error In 3.3 and later, this is now an alias for OSError. I hacked up the server code enough that I was able to trigger the exception handler in server.py from test-http-bundle1.t. Other instances of this either subscript through the `args` member, or reference the errno or strerror attributes. Note that on Windows, the errno value seems to reflect the Winsock error, so the various tests for EPIPE seem like they would always fail. But that seems to be the case in py2 as well.

test: update test-logtoprocess.t to not requires less The test previously assumed that a working pager was present in the test environment. Update it to use the fakepager instead. Differential Revision: https://phab.mercurial-scm.org/D5399

py3: quote several instances of $PYTHON for Windows Python3 lives in Program Files by default. The last remaining unquoted instance is in test-hghave.t. I can't figure out how to quote that, as it either complains: 'c:\\Program' is not recognized as an internal or external command, operable program or batch file. or $TESTTMP.sh: line 22: "C:/Program Files/Python37/python.exe": $ENOENT$

py3: adapt test-largefiles-wireproto.t to python3

py3: byteify test-push-race.t

py3: convert filename to bytes in test-hardlinks.t Windows was complaining about passing str to CreateFileA() in win32.py.

narrow: keep bookmarks temporarily stripped for as long as commits are The narrow extension also has support for shallowness and for inserting older commits on pull. It works by temporarily stripping newer commits, adding the older commits, then re-applying the stripped bundle. The regular Mercurial server uses that when you widen, although it shouldn't be necessary there. Our Google-internal server does it when the user requests an older commit. Our Google-internal tests fail since 7caf632e30c3 (filecache: unimplement __set__() and __delete__() (API), 2018-10-20). I haven't quite understood the problem, but it's related to the way we temporarily hide bookmarks while the commits they point to are stripped. When a transaction is started, Mercurial tries to read various things from the repo for the transaction summary. That leads to computation of hidden commits, which leads to an attempt to find commits pinned by bookmarks. This is the reason we temporarily hide the bookmarks. I think the aforementioned commit makes the restored bookmarks visible earlier than before (which seems like an improvement), so we can no longer incorrectly rely on the repo._bookmarks field being cached too long (IIUC). This patch makes it so we restore the temporarily hidden bookmarks only after the temporary bundle has been re-applied. It also adapts the code to update the repo.__bookmarks field using the pattern described in the aforementioned commit instead of writing directly to the fiels. I have spent many hours trying to understand what was going on here, but I still don't know if this can also happen without our custom server. So this patch unfortunately does not add any tests; I have only been able to test the fix using our Google-internal tests. Differential Revision: https://phab.mercurial-scm.org/D5398

narrow: drop obsolete support for old Mercurial Not needed since narrow is shipped with core. Differential Revision: https://phab.mercurial-scm.org/D5397

tests: test that narrow preserves bookmarks on widen When widening inserts older commits in the changelog, we have to preserve bookmarks so they are not removed by the call to repair.strip() we temporarily do. We didn't have any tests for that before. Differential Revision: https://phab.mercurial-scm.org/D5396

rebase: clarify that commits that become empty are skipped Our message just said that it created no changes, but we didn't explicitly say that that meant that the the commit was not rebased. It also wasn't clear why it created no changes, so this patch also clarifies that that was because the changes were already in the destination. Differential Revision: https://phab.mercurial-scm.org/D5395

rebase: format commit in usual way when explaining that it became empty We use the _ctxdesc() for other similar messages, so it seems we should use it here too. Differential Revision: https://phab.mercurial-scm.org/D5394

rust: propagate error of index_get_parents() properly Before, rustla_contains() would return 0 on error, and the exception would be cleared or noticed somewhere else. We need to propagate the error from AncestorsIterator up to the FFI surface.

rust: look up HgRevlogIndex_GetParents() from symbol table And removes the unused index_get_parents_checked() function. I expect the Index struct will be turned into a pyobject type, though I haven't written any PoC-level patches yet.

revlog: add public CPython function to get parent revisions Since this is a public function, it validates the input revision, and supports nullrev. index_get_parents_checked() will be replaced by this function.

revlog: rename indexType to HgRevlogIndex_Type as it's a global symbol It follows the CPython naming convention, but uses Hg instead of Py.

revlog: export symbol of indexType The idea is to wrap the index object with rust-cpython. I haven't tried it, but it should be doable. We'll probably need a better interface than raw function pointers to do more in Rust.

setup: fix path of rust source dependencies Appears that they were screwed up while moving the source files around. Since rust-direct-ffi sources are automatically added to the depends list, we only need to list hg-core sources.

rebase: remove now-unnecessary arguments to _abort() Since _abort() is now an instance method, we don't need to pass state that's available on "self". Differential Revision: https://phab.mercurial-scm.org/D5393

rebase: move abort() onto rebaseruntime The function depends on a lot of the state of rebaseruntime, so it makes sense for it to be an instance method. This will let us remove many of the arguments to the method. That will be done in a later patch. Differential Revision: https://phab.mercurial-scm.org/D5392

cleanupnodes: trust caller when "moves" is not None If "moves" (indicating how to move bookmarks) is None, we fill it out based on "replacements" (indicating which obsmarkers to add). If "moves" is not None, we would still add items based on "replacements". This makes it impossible to pass "moves={}" and not move bookmarks, which surprised me. The only caller that currently passes a value for "moves" was the rebase extension and there we were already adding bookmark moves corresponding to obsmarker additions, so it should not be impacted. Differential Revision: https://phab.mercurial-scm.org/D5391

shelve: change transaction description from "commit" to "shelve" "commit" was probably a copy&paste mistake. Differential Revision: https://phab.mercurial-scm.org/D5390

shelve: drop unnecessary backup of dirstate for phase-based case Regular shelve has a hack using an uncommitted transaction that's then aborted at the end of the operation. It preserves the dirstate across the abort, however, by saving a backup copy of it. Phase-based shelve instead commits the transaction, so the hack shouldn't be necessary there. Differential Revision: https://phab.mercurial-scm.org/D5389

tests: split test-shelve.t in two test-shelve.t dominated run time for all shelve tests. Before: # Ran 9 tests, 1 skipped, 0 failed. real 0m43.568s user 2m15.822s sys 0m40.857s After: # Ran 11 tests, 1 skipped, 0 failed. real 0m24.574s user 2m21.354s sys 0m40.435s Differential Revision: https://phab.mercurial-scm.org/D5388

remotefilelog: rely on progress helper for keeping track of position The progress helper class keeps track of its current position, so we don't need a "count" variable for that. Differential Revision: https://phab.mercurial-scm.org/D5386

remotefilelog: reduce use of "count" container We had already given the initial value of "count[0]" a name, so just use that when applicable. Differential Revision: https://phab.mercurial-scm.org/D5385

remotefilelog: replace a "a=[expr]; b=a[0]" by "b=expr; a = [b]" Simpler is better. Differential Revision: https://phab.mercurial-scm.org/D5384

remotefilelog: remove an unnecessary update of "count" container The "count" container is never used after this point, so there is no need to update it. Differential Revision: https://phab.mercurial-scm.org/D5383

remotefilelog: avoid temporarily using "count" variable as synonym for "total" The "count" variable is generally used for updating progress, but early in fileserverclient.request(), its used to mean the total count. We already have another "total" variable for that, so it seems much clearer to use that. Differential Revision: https://phab.mercurial-scm.org/D5382

remotefilelog: use progress helper in fileserverclient Differential Revision: https://phab.mercurial-scm.org/D5381

remotefilelog: use progress helper in shallowrepo Differential Revision: https://phab.mercurial-scm.org/D5380

remotefilelog: use progress helper in shallowbundle Differential Revision: https://phab.mercurial-scm.org/D5379

remotefilelog: use progress helper in repack Differential Revision: https://phab.mercurial-scm.org/D5378

remotefilelog: use progress helper in remotefilelogserver Differential Revision: https://phab.mercurial-scm.org/D5377

remotefilelog: use progress helper in basestore Differential Revision: https://phab.mercurial-scm.org/D5376

remotefilelog: use progress helper when analyzing repos Differential Revision: https://phab.mercurial-scm.org/D5375

fastannotate: use progress helper Differential Revision: https://phab.mercurial-scm.org/D5374

debugbuildannotatecache: use progress helper Differential Revision: https://phab.mercurial-scm.org/D5373

upgrade: clarify "aggressivemergedelta" handling We rename "aggressivemergedelta" argument to "forceaggressivemergedelta". The previous argument naming could infer an absolute control on the behavior. However, the code show we respect the config option if set.

upgrade: test that fncache is valid after repository upgrade The behavior was correct. Now it is correct and tested.

tests: update `rm` invocation for new location of checklink This should fix the test on FreeBSD. Differential Revision: https://phab.mercurial-scm.org/D5387

bookflow: new extension for bookmark-based branching This extension should be helpful for feature branches - based workflows. At my company we first considered branches, but weren't sure about creating a lot of permanent objects. We tried bookmarks, but found some scenarios to be difficult to control. The main problem, was the active bookmark being moved on update. Disabling that made everything a lot more predictable. Bookmarks move on commit, and updating means switching between them. The extension also implements a few minor features to better guide the workflow: - hg bookmark NAME can be ambiguous (create or move), unlike hg branch. The extension requires -rev to move. - require an active bookmark on commit. - some bookmarks can be protected (like @), useful for teams, that require code reviews. - block creation of new branches. The initial implementation requires no changes in the core, but it does rely on some implementation details. I thought it may be useful to discuss the functionality first, and then focus making the code more robust. Differential Revision: https://phab.mercurial-scm.org/D4312

rust: make clean takes care of rust/target This isn't the prettiest way of doing it, but it doesn't require looking up cargo, or wondering whether that should be part of setup.py clean. Differential Revision: https://phab.mercurial-scm.org/D5369

rust: rename local variables in AncestorsIterator::next It was confusing to have p1 and parents.1 ; (p1, p2) is clearer. Differential Revision: https://phab.mercurial-scm.org/D5365

tests: stabilize test-inherit-mode.t on FreeBSD and macOS again This is the same fix as 90e26ef4cbb1, just repeated on the new file location. Differential Revision: https://phab.mercurial-scm.org/D5371

tests: drop redundant "#if execbit" from test-rebase-inmemory.t The whole file is already guarded by "#require execbit". Differential Revision: https://phab.mercurial-scm.org/D5366

narrow: don't resurrect old commits when narrowing (don't strip obsmarkers) If you have an old obsolescence-chain of commits that has been pruned and you narrow your repo so that some of those commits get stripped (because they affected the removed paths), then we would currently resurrect the commit that came before (along the obsmarker chain) the last stripped commit. That happens by the usual rules for obsmarker-stripping. However, it's quite surprising when it happens when you narrow your repo. This patch makes narrowing not strip obsmarkers. Differential Revision: https://phab.mercurial-scm.org/D5364

commandserver: turn server debug messages into logs They were ui.debug() just because commandserver.log() was noop if no client connected.

commandserver: add config knob for various logging options The default rotating options are copied from the blackbox extension.

commandserver: expand log path for convenience This allows us to set the log path relative to $XDG_RUNTIME_DIR, for instance. [cmdserver] log = $XDG_RUNTIME_DIR/chg/server.log

commandserver: switch logging facility to ui.log() interface The "pager subcommand" message is removed since ui isn't accessible there. I think that's okay as cmdtable[cmd]() will call attachio() and some debug message will be printed.

commandserver: install logger to record server events through canonical API The global commandserver.log() will be replaced with this.

commandserver: enable logging when server process started This allows us to keep track of server events before client connects to the server. Tests will be added later. Currently there's no log() call to check if things are working well.

test-commandserver: change way of triggering early crash Future patches will move the logging facility out of the server class, so cmdserver.log can't be (ab)used for this purpose. Instead, let's hook the factory function to raise exception.

loggingutil: add basic logger backends These classes will be used in command server. They are similar to the blackboxlogger, but it can't be factored out since the blackbox is so tightly coupled with a repo object.

hgweb: register web.comparisoncontext to the config table This was caught in some server side logging added to debug py3 issues.

merge with stable

Added signature for changeset 1c8c54cf9725

Added tag 4.8.1 for changeset 1c8c54cf9725

rebase: fix path auditing to audit path relative to repo root (issue5818) Before this patch, when rebasing a file called "foo/bar", we would check e.g. if "/foo" (i.e. rooted at the file system root) was a symlink. Differential Revision: https://phab.mercurial-scm.org/D5361

tests: show bad path auditing in in-memory rebase Thanks to Yuya for providing this test case in https://bz.mercurial-scm.org/show_bug.cgi?id=5818. Differential Revision: https://phab.mercurial-scm.org/D5368

tests: add a missing "cd .." to test-rebase-inmemory.t Differential Revision: https://phab.mercurial-scm.org/D5367

rust: fix possible out-of-bounds read through index_get_parents() index_get_parents() is an internal function, which doesn't check if the specified rev is valid. If rustlazyancestors() were instantiated with an invalid stoprev, it would access to invalid memory region. This is NOT a security fix as there's no Python code triggering the bug, but included in this series to not give a notion about the memory issue fixed by the previous patch.

revlog: fix out-of-bounds access by negative parents read from revlog (SEC) 82d6a35cf432 wasn't enough. Several callers don't check negative revisions but for -1 (nullrev), which would directly lead to out-of-bounds read, and buffer overflow could follow. RCE might be doable with carefully crafted revlog structure, though I don't think this would be useful attack surface.

rust: peek_mut optim for lazy ancestors This is one of the two optimizations that are also present in the Python code: replacing pairs of pop/push on the BinaryHeap by single updates, hence having it under the hood maintain its consistency (sift) only once. On Mozilla central, the measured gain (see details below) is around 7%. Creating the PeekMut object by calling peek_mut() right away instead of peek() first is less efficient (gain is only 4%, stats not included). Our interpretation is that its creation has a cost which is vasted in the cases where it ends by droping the value (Peekmut::pop() just does self.heap.pop() anyway). On the other hand, the immutable peek() is very fast: it's just taking a reference in the underlying vector. The Python version still has another optimization: if parent(current) == current-1, then the heap doesn't need to maintain its consistency, since we already know that it's bigger than all the others in the heap. Rust's BinaryHeap doesn't allow us to mutate its biggest element with no housekeeping, but we tried it anyway, with a copy of the BinaryHeap implementation with a dedicaded added method: it's not worth the technical debt in our opinion (we measured only a further 1.6% improvement). One possible explanation would be that the sift is really fast anyway in that case, whereas it's not in the case of Python, because it's at least partly done in slow Python code. Still it's possible that replacing BinaryHeap by something more dedicated to discrete ordered types could be faster. Measurements on mozilla-central: Three runs of 'hg perfancestors' on the parent changeset: Moyenne des médianes: 0.100587 ! wall 0.100062 comb 0.100000 user 0.100000 sys 0.000000 (best of 98) ! wall 0.135804 comb 0.130000 user 0.130000 sys 0.000000 (max of 98) ! wall 0.102864 comb 0.102755 user 0.099286 sys 0.003469 (avg of 98) ! wall 0.101486 comb 0.110000 user 0.110000 sys 0.000000 (median of 98) ! wall 0.096804 comb 0.090000 user 0.090000 sys 0.000000 (best of 100) ! wall 0.132235 comb 0.130000 user 0.120000 sys 0.010000 (max of 100) ! wall 0.100258 comb 0.100300 user 0.096000 sys 0.004300 (avg of 100) ! wall 0.098384 comb 0.100000 user 0.100000 sys 0.000000 (median of 100) ! wall 0.099925 comb 0.100000 user 0.100000 sys 0.000000 (best of 98) ! wall 0.133518 comb 0.140000 user 0.130000 sys 0.010000 (max of 98) ! wall 0.102381 comb 0.102449 user 0.098265 sys 0.004184 (avg of 98) ! wall 0.101891 comb 0.090000 user 0.090000 sys 0.000000 (median of 98) Mean of the medians: 0.100587 On the present changeset: ! wall 0.091344 comb 0.090000 user 0.090000 sys 0.000000 (best of 100) ! wall 0.122728 comb 0.120000 user 0.110000 sys 0.010000 (max of 100) ! wall 0.093268 comb 0.093300 user 0.089300 sys 0.004000 (avg of 100) ! wall 0.092567 comb 0.100000 user 0.090000 sys 0.010000 (median of 100) ! wall 0.093294 comb 0.080000 user 0.080000 sys 0.000000 (best of 100) ! wall 0.144887 comb 0.150000 user 0.140000 sys 0.010000 (max of 100) ! wall 0.097708 comb 0.097700 user 0.093400 sys 0.004300 (avg of 100) ! wall 0.094980 comb 0.100000 user 0.090000 sys 0.010000 (median of 100) ! wall 0.091262 comb 0.090000 user 0.080000 sys 0.010000 (best of 100) ! wall 0.123772 comb 0.130000 user 0.120000 sys 0.010000 (max of 100) ! wall 0.093188 comb 0.093200 user 0.089300 sys 0.003900 (avg of 100) ! wall 0.092364 comb 0.100000 user 0.090000 sys 0.010000 (median of 100) Mean of the medians is 0.0933 Differential Revision: https://phab.mercurial-scm.org/D5358

fuzz: grep away HAVE_GETC_UNLOCKED in pyconfig.h to avoid msan badness Per discussion with Greg Smith and the patches on https://bugs.python.org/issue35214. This, combined with the previous patch, fixes msan builds on oss-fuzz. Differential Revision: https://phab.mercurial-scm.org/D5363

fuzz: more correctly specify CFLAGS and LDFLAGS when building Python Gets us closer to a working msan build alongside our asan build. Differential Revision: https://phab.mercurial-scm.org/D5362

tests: stabilize test-blackbox.t on Windows I didn't look into why the error is more detailed, but that seems like it's a good thing (other than for recording tests).

tests: stabilize for recent wcache changes This goes with 47e3f554df35::d5622dfe4ba3. I'm not sure if it was really expected that there would be no wcache directory if neither execbit nor symlink is supported.

extdiff: avoid double backslashes in the displayed tool path on Windows This shows the tool path in the help, and changed in 67b180c0e263. uirepr() already does the same thing, but that undoes the mangling in its call to repr().

contrib: add a helper script that help to build interesting repositories The script is dedicated to building a couple of repositories that should be interesting to run discovery from one another. It seems a common enough need to contribute it upstream.

py3: listify filter() to call len() on it Differential Revision: https://phab.mercurial-scm.org/D5354

rebase: fix dir/file conflict detection when using in-mem merge Differential Revision: https://phab.mercurial-scm.org/D5360

tests: show that in-mem rebase does not find path dir/file conflicts Differential Revision: https://phab.mercurial-scm.org/D5359

extdiff: register the configuration generated commands with a help category Otherwise, 'extdiff' shows up under file management and the rest of the commands are at the bottom under 'Uncategorized'.

loggingutil: document openlogfile() This function will be used later for command-server logging.

loggingutil: extract openlogfile() and proxylogger to new module This module isn't placed under the "utils" package since it needs "ui" to process things. It's called "loggingutil", not "logutil" because the word "log" is too obscure in our codebase.

blackbox: pass in options to _openlogfile() as arguments This prepares for extracting utility function from the blackbox module.

blackbox: just try writing to repo.vfs and update lastlogger on success This is simpler and more robust. Before, an empty ".hg" directory would be created if it's removed after checking vfs.isdir('.').

vfs: add option to not create parent directories implicitly In blackbox, we don't want to create a ".hg" directory by mistake. This provides a race-safe option to achieve that.

repo: add a `wcachevfs` to access the `.hg/wcache/` directory This wvfs will allow us to migrate various cache to the new `wcache` directory. Helping with cache issues with "share".

cache: create `wcache` directory at init time The cache directory will be needed very quickly, so it seems simpler to create it early to make sure it has the same owner and permission than the other directory in the repository.

cache: create `cache` directory at init time The cache directory will be needed very quickly, so it seems simpler to create it early to make sure it has the same owner and permission than the other directory in the repository.

check-exec: write file in 'wcache' instead of 'cache' Some cache are relevant or affected by the working copy used. So the `.hg/cache` directory is not the best place for them because multiple shared repository can end up fighting over them. To address this issue, we introduce a new 'wcache' directory to host this kind of cache. The first user are the `checkisexec` type file. These files describe property of the working copy and fit the use-case well.

rebase: abort in-mem rebase if there's a dirty merge state In-memory merge uses the on-disk merge state, so we should not allow it run in-memory merge when the merge state is not clean. We should probably not use the on-disk merge state when running in-memory merge, but chaning that is not suitable for the stable branch. Differential Revision: https://phab.mercurial-scm.org/D5357

rebase: preserve working copy when redoing in-mem rebase on disk When in-memory rebase runs into conflicts, we retry it on disk. But before we do that, we abort the in-memory rebase. That is done because even though it's mostly in memory, there are still a few state files written (e.g. the merge state). We should make it not write those files so we don't need to abort, but for the stable branch, let's explicitly clear the state we need to clear instead of running the usual abort code. Differential Revision: https://phab.mercurial-scm.org/D5356

tests: show that in-mem rebase falling back loses state Both working copy changes and the merge state is lost when in-memory rebase falls back to on-disk mode. Differential Revision: https://phab.mercurial-scm.org/D5355

commandserver: get around ETIMEDOUT raised by selectors2 selector.select() should exits with an empty event list on timed out, but selectors2 raises OSError if timeout expires while recovering from EINTR. Spotted while debugging new chg feature.

selectors2: backport minimal fix of timeout handling from 2.0.1 The original code would raise TypeError since OSError() doesn't support keyword arguments. We can't simply import the selectors 2.0.1, which still spawns "uname -p" through platform.system(). We could switch to the unreleased version, but I decided to not right now to minimize the change.

mmapindex: set default to 1MB mmapping index is more efficient if we only need a small part of it. The 1MB value has been picked arbitrarily, a lower value might be better. On a large repository with a 60MB index, we see the following performance gain: hg perfindex before: ! wall 0.032023 comb 0.040000 user 0.000000 sys 0.040000 (best of 100) after: ! wall 0.000196 comb 0.000000 user 0.000000 sys 0.000000 (best of 1060) The speed boost benefit all cases, including the one where the full index needs to be parsed. hg perfindex --rev 0 before: ! wall 0.040673 comb 0.030000 user 0.000000 sys 0.030000 (best of 100) after ! wall 0.010713 comb 0.020000 user 0.010000 sys 0.010000 (best of 212) This gain reflect in higher level operation: hg perfbookmarks --clear-revlogs before: ! wall 0.161339 comb 0.160000 user 0.130000 sys 0.030000 (best of 56) after: ! wall 0.123228 comb 0.120000 user 0.120000 sys 0.000000 (best of 68)

mmapindex: move the 'mmapindexthreshold' option out of experimental The option is useful and should be advertised more. We move it out of experimental as a first step. The `storage` section is selected as this is related to how the storage is accessed. A new 'performance' section might be more appropriate. We move from 'mmapindexthreshold` to `mmap-threshold` as non-index item are also suitable for mmap (eg: the rev-branch-cache). If relevant, we can introduce sub-option `mmap-threshold.revlog-index` later.

perf: add a --rev attribute to perfindex This allow for benchmarking the time necessary to look for other version than the tip.

perf: update perfindex to be more realistic The previous code was creating a revlog manually, we now use the actual `localrepo` method to create it. We have to jump though extra hops to work around the impact of filecache.

match: drop unnecessary wrapping of regex in group It seems the regexes have been wrapped in an unnamed group since b6c42714d900 (Add locate command., 2005-07-05). In that commit, the grouping was needed because there was a "head" ('^') added before the group and a "tail" (os.sep) added after it. It seems the head was moved inside the group in 1c0c413cccdd (Get add and locate to use new repo and dirstate walk code., 2005-07-18) and the tail was moved inside the group in 89985a1b3427 (Clean up walk and changes code to use normalised names properly., 2005-07-31), So it seems to me that we've carried around the unnecessary group for 13 years. This patch removes it. Differential Revision: https://phab.mercurial-scm.org/D5352

match: use _BASE_SIZE instead of magic value 4 Differential Revision: https://phab.mercurial-scm.org/D5351

match: make "groupsize" include the trailing "|" I think this is a little easier to follow and it will simplify later patches too. Differential Revision: https://phab.mercurial-scm.org/D5350

match: fix an unaligned (but harmless) indent Differential Revision: https://phab.mercurial-scm.org/D5349

match: raise an Abort error instead of OverflowError This case of OverflowError (one single pattern being too large) has never been properly caught in the past.

match: avoid translating glob to matcher multiple times for large sets For hgignore with many globs, the resulting regexp might not fit under the 20K length limit. So the patterns need to be broken up in smaller pieces. Before this change, the logic was re-starting the full process from scratch for each smaller pieces, including the translation of globs into regexp. Effectively doing the work over and over. If the 20K limit is reached, we are likely in a case where there is many such glob, so exporting them is especially expensive and we should be careful not to do that work more than once. To work around this, we now translate glob to regexp once and for all. Then, we assemble the resulting individual regexp into valid blocks. This raises a very significant performance win for large `.hgignore file`: Before: ! wall 0.153153 comb 0.150000 user 0.150000 sys 0.000000 (median of 66) After: ! wall 0.059793 comb 0.060000 user 0.060000 sys 0.000000 (median of 100)

match: extract function that group regexps

match: test for overflow error in pattern If a single pattern is too large to handle, we raise an exception. This case is now doctested.

match: extract a literal constant into a symbolic one

tests: apply binary mode to output in seq.py I noticed this when playing with running tests using WSL, and iterating over the output yielded '0\r', '1\r',... Most of the other *.py tools do this, and `seq` on MSYS lacks '\r' in the output, so this is more consistent.

perf: add a `--clear-caches` to `perfbranchmapupdate` This flag will help to measure the time we spend loading various cache that support the branchmap update. Example for an 500 000 revisions repository: hg perfbranchmapupdate --base 'not tip' --target 'tip' ! wall 0.000860 comb 0.000000 user 0.000000 sys 0.000000 (best of 336) hg perfbranchmapupdate --base 'not tip' --target 'tip' --clear-caches ! wall 0.029494 comb 0.030000 user 0.030000 sys 0.000000 (best of 100)

perf: start from an existing branchmap if possible If the --base set if a superset of one of the cached branchmap, we should use as a starting point. This greatly help the overall runtime of `hg perfbranchmapupdate` For example, for a repository with about 500 000 revisions, using this trick make the command runtime move from about 200 second to about 10 seconds. A 20x gain.

perf: rely on repoview for perfbranchmapupdate Using 'repoview' matching the base and target subset make the benchmark more realistic. It also unlocks optimization to make the command initialization faster.

perf: pre-indent some code in `perfbranchmapupdate` This make the next patch easier to read.

perf: add a `perfbranchmapupdate` command This command benchmark the time necessary to update the branchmap between two sets of revisions. This changeset introduce a first version, doing nothing fancy regarding cache or other internal details.

push: config option to control behavior when pushing to a publishing server Pushing to a publishing server by mistake can lead to a difficult situation to solve because evolution doesn't work on public changesets. This new experimental config tries to help avoiding unintentionally (or at least being aware of) pushing to publishing remotes. `hg push --publish` can be used to make push succeed even when auto-publish is set to 'abort'.

narrowcommands: remove an unrequired `repo.narrowpats` call We call that few lines above and do nothing significant in between which can change the narrowpats. So let's use values returned by that call. Differential Revision: https://phab.mercurial-scm.org/D5348

manifest: reject lines shorter than 42 bytes, not 22 Yuya correctly spotted during the review of f27f8e9ef1e73 that we're dealing with hexlified hashes here, and so it should be 42 bytes not 22. Differential Revision: https://phab.mercurial-scm.org/D5347

blackbox: initialize logger with repo instance The blackboxlogger is unusable without a repo. Let's simply initialize it with a repo instance.

blackbox: do not nullify repo to deactivate the logger on failure The _repo will be a mandatory attribute. Instead, make the logger to not track any events.

blackbox: extract global last logger to proxylogger class So the blackboxlogger can be instantiated with a repo.

ui: pass in bytes opts dict to logger.log() This is the convention of the Mercurial API.

ui: pass in formatted message to logger.log() This makes sure that all logger instances will handle the message arguments properly.

blackbox: send debug message to logger by core ui Since the core ui.log() may recurse into ui.log() through ui.debug(), it must guard against recursion. The ui extension class can finally be removed.

blackbox: change the way of deactivating the logger on write error This prepares for the upcoming code move. The recursion guard will be ported to the core ui.

match: remove obsolete catching of OverflowError Since 0f6a1bdf89fb (match: handle large regexes, 2007-08-19), we catch an OverflowError from the regex engine and split up the regex if that happens. In 59a9dc9562e2 (ignore: split up huge patterns, 2008-02-11), that was extended to raise an OverflowError in our code even if the regex engine doesn't raise it. It's unclear if there was a range of regex sizes where the OverflowError would be raised from the regex engine but that were still below the limit we added in our code. Either way, both limitations were probably removed in Python 2.7.4 when the regex code width was extended from 16bit to 32bit (or Py_UCS4) integer (thanks to Yuya for finding that out). If at least the first limitation was removed, we no longer should be using OverflowError for flow control, so this patch changes that. Differential Revision: https://phab.mercurial-scm.org/D5309

sparse: raise a move verbose index error from the C code If we don't like a value we should print it.

narrow: drop the bundle2 capability since we have server capabilities (BC) This patch drops the narrow bundle2 capabilities since we introduced narrow server capabilities which are more nice and now used everywhere. I am not sure what it can affect, so on safe side I marked this as BC. Also I removed the NARROWCAP constant as that kind of conflicts with the same name constant in wireprototypes.py. Differential Revision: https://phab.mercurial-scm.org/D4892

vfs: extract the audit path logic into a submethod This will make it possible to apply it in more cases.

subrepo-git: use an official origvfs when appropriate The origvfs has the auditor properly set and can move file without issue. The current code is currently working without errors because rename are not audited, yet.

revert: extract origvfs logic in a sub-function The subrepo's "revert" logic could benefit from it.

vfs: treat 'undo.' file the same as 'journal.' file They are the same kind of file, they are protected by the store lock, but directly lives inside the '.hg' directory. No warnings were ever raised about them because `vfs.rename` is not audited. Something we are trying to change.

perf: add a perfignore command The command is meant to benchmark operations related to hgignore. Right now the command is benchmarking the loading time of the hgignore rules.

py3: use pycompat.xrange instead of xrange xrange does not exists on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5302

store: write fncache only once if there are both adds and removes Differential Revision: https://phab.mercurial-scm.org/D5307

perf: disable revlogs clearing in `perftags` by default This aligns things with what `perfbookmarks` does. I decided to disable the revlogs clearing by default to focus on the core logic by default, ignoring side effects. If we prefer to emphasize the side effect, we can instead keep this on in `perftags` and enable it by default in `perfbookmarks`.

perf: add a `clear-revlogs` flag to `perfbookmarks` This flag (off by default) makes it possible to enable the refresh of the changelog and revlog. This is useful to check for costly side effects of bookmark loading. Usually, these side effects are shared with other logics (eg: tags). example output in my mercurial repo (with 1 bookmark, so not a great example): $ hg perfbookmarks ! wall 0.000044 $ hg perfbookmarks --clear-revlogs ! wall 0.001380

tags: cache `repo.changelog` access when checking tags nodes The tags reading process checks if the nodes referenced in tags exist. Caching the access to `repo.changelog` provides a large speedup for repositories with many tags. running `hg perftags` in a large private repository before: ! wall 0.393464 comb 0.390000 user 0.330000 sys 0.060000 (median of 25) after: ! wall 0.267711 comb 0.270000 user 0.210000 sys 0.060000 (median of 38)

perf: add a `clear-revlogs` flag to `perftags` This flag (on by default) makes it possible to disable the refresh of the changelog and revlog. This is useful to check for the time spent in the core tags logic without the associated side effects. Usually, these side effects are shared with other logics (eg: bookmarks). Example output in my Mercurial repository $ hg perftags ! wall 0.017919 comb 0.020000 user 0.020000 sys 0.000000 (best of 141) $ hg perftags --no-clear-revlogs ! wall 0.012982 comb 0.010000 user 0.010000 sys 0.000000 (best of 207)

perf: stop creating new revlog by hand in perftags It's better to let the repository logic create its own object. We now just clear the cache. New object will be automatically created from there.

revlog: update the documentation for `trim_endidx` The function role drifted since the function was commented.

revlog: properly detect corrupted revlog in `index_get_length` Pointed out by Yuya Nishihara.

perf: rename `perfhelper-tracecopies` to `perfhelper-pathcopies` The command it supports is called `perfpathcopies`. It seems better to align the names.

perf: add a docstring to `perfpathcopies` This will help people to find this command.

revlog: update the docstring of `ancestors` to match reality Code using this method expect the revision to be (reverse) sorted. As pointed by Yuya Nishihara, the docstring should reflect that.

remotefilelog: fix typo in docstring Differential Revision: https://phab.mercurial-scm.org/D5306

store: append to fncache if there are only new files to write Before this patch, if we have to add a new entry to fncache, we write the whole fncache again which slows things down on large fncache which have millions of entries. Addition of a new entry is common operation while pulling new files or commiting a new file. This patch adds a new fncache.addls set which keeps track of the additions happening and store them. When we write the fncache, we will just read the addls set and append those entries at the end of fncache. We make sure that the entries are new entries by loading the fncache and making sure entry does not exists there. In future if we can check if an entry is new without loading the fncache, that will speed up things more. Performance numbers for commiting a new file: mercurial repo before: 0.08784651756286621 after: 0.08474504947662354 mozilla-central before: 1.83314049243927 after: 1.7054164409637451 netbeans before: 0.7953150272369385 after: 0.7202838659286499 pypy before: 0.17805707454681396 after: 0.13431048393249512 In our internal repo, the performance improvement is in seconds. I have used octobus's ASV perf benchmark thing to get the above numbers. I also see some minute perf improvements related to creating a new commit without a new file, but I believe that's just some noise. Differential Revision: https://phab.mercurial-scm.org/D5301

py3: fix couple of division operator to do integer divison Differential Revision: https://phab.mercurial-scm.org/D5305

py3: use dict.items() instead of dict.iteritems() dict.iteritems() does not exist on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5304

py3: convert strings to bytes in tests/test-remotefilelog-histpack.py Differential Revision: https://phab.mercurial-scm.org/D5303

py3: replace str() with pycompat.bytestr() or ('%d' % int) After this patch, test-fastannotate-diffopts.t is about to pass. There are some extra newlines in the output. Differential Revision: https://phab.mercurial-scm.org/D5300

perf: add a `--timing` argument to `perfhelper-tracecopies` The new argument will help picking better pair for benchmarking. See documentation for details.

ui: manage logger instances and event filtering by core ui The setup code in blackbox needs more tweaks since it has lots of black magics. I'll fix them by follow-up patches. To be clear, the goal of this series is to provide a proper way for command server to install its own logger. I need it to debug in-memory repository cache.

extensions: add "uipopulate" hook, called per instance, not per process In short, this is the "reposetup" function for ui. It allows us to modify ui attributes without extending ui.__class__. Before, the only way to do that was to abuse the config dictionary, which is copied across ui instances. See the next patch for usage example.

hgweb: load globally-enabled extensions explicitly Before, extensions were loaded as a side effect of hg.repository() if the hgweb was executed as a CGI/WSGI. I want to make it explicit so that another ui hook can be inserted after extensions.loadall().

tests: sniff for libfuzzer actually being available in test-fuzz-targets.t When I upgraded the FreeBSD buildbot to 11.2 it seems we picked up clang6, but the default clang on FreeBSD doesn't include libfuzzer. I can't find a way to sniff for libfuzzer without running a compile, so here we are. Differential Revision: https://phab.mercurial-scm.org/D5270

tests: sniff for /usr/local/bin/gmake and use it in test-fuzz-targets.t This isn't as robust as it probably should be, but for now it'll get the job done on the buildbots. Differential Revision: https://phab.mercurial-scm.org/D5269

tests: stabilize test-inherit-mode.t on FreeBSD and macOS (issue6026) Symbolic links are funny permissions-wise, but on the linked issue Yuya has convinced me that we can ignore this permissions issue on macOS (FreeBSD allows setting permissions bits but ignores them) and we'll be in fine shape.

wireprotov2peer: wait for initial object before resolving future As part of rolling out wireprotov2 with redirect support, I encountered an edge case with regards to future resolution. Essentially, the initial response frame from the server did not fully decode the initial CBOR object. The frame wasn't marked as EOS. In the previous code, we resolved the future for the request to response.objects(), which mapped to the commandresponse instance which would eventually produce a redirect. Upon receiving subsequent data, the initial CBOR object containing the redirect would be decoded and we'd process the redirect. However, the future would already have been resolved with the initial commandresponse.objects() and the client iterating over the objects wouldn't receive any objects from the redirect because the redirect was populating a different commandresponse instance! This commit changes the logic so we don't resolve futures until the initial CBOR response object is fully decoded or until EOS occurs. In cases where there is an empty or partial frame associated with a redirect, the future will now resolve with the commandresponse containing the proper series of decoded objects.

wireprotov2peer: always return a bool from _processredirect() Without this, we may stop servicing the redirect response if the future has already been resolved. And the future will often be resolved very early, since many consumers iterate the decoded CBOR object stream and expect data to lazily arrive.

tests: disable remotefilelog on Windows I've spent a non trivial amount of time trying to eliminate the test errors, but it's looking like this is pretty dependent on Unix support. For example, there are attempts to delete open files, and uses of threads that report I/O attempts on closed files. (Maybe this is a race condition? Don't we usually use processes as workers on Windows?) In any event, I don't want real new errors elsewhere to be masked by these known problems. For some reason $CACHEDIR is reported as missing in test-remotefilelog-repack.t, but it actually exists in the hgcloneshallow call inside shallowutil.mkstickygroupdir(). By the time the process exits, it's gone. I don't see it being removed by code that calls 'rmdir' or 'remove' in the extension itself.

perf: run 'setup' function during stub run The benchmarked function might need the content of the setup to be run in order to function properly.

perf: fallback to subset if ondisk cache is missing in perfbranchmapload If there is no branchmap on disk for that filter, it means that the cache from some subset's filter is relevant for this one. We look for it instead of aborting. That way it is much simpler to run the command in an automated way. We can now add it to `test-contrib-perf.t`.

perf: prewarm the branchmap in perfbranchmapload It is not very interesting to have the command randomly failing because the branchmap for the tested filter happens to be cold. So we make sure to have a valid up to date branchmap before going further. The data might still be missing from disk if a subset was equivalent. See next changeset for details and fix.

localrepo: correct docstring of filectx() The same reason as b6c2543e1dd8. It can't be any changeset specifiers but revision number.

annotate: remove dead code to not convert path to relative path It's annotate. There should be at least one file path specified.

annotate: start pager soon after command options are validated It helps extracting helper class.