tests: delete unused function in test-rename-merge2.t Differential Revision: https://phab.mercurial-scm.org/D6253

tests: make merge conflicts explicit in `hg annotate` tests We were using `true` as merge tool. I think it makes the test easier to understand if we make the conflicts explcit. It also papered over a conflict that shouldn't have been a conflict (just a bug in copy tracing). I've marked that "BROKEN". Differential Revision: https://phab.mercurial-scm.org/D6252

narrow: make warning about possibly dirty files respect ui.relative-paths Differential Revision: https://phab.mercurial-scm.org/D6264

merge: forgot to pull before release These two changes should be part of 5.0, but we are fine leaving them out of the RC.

Added signature for changeset 4a8d9ed86475

Added tag 5.0rc0 for changeset 4a8d9ed86475

merge: default into stable for release candidate

debugdiscovery: include the number of heads in all sets We already displayed information about heads of the common set that are either local or remote heads. We now also do so for heads of the common set that are both local and remote heads too. This is useful because various step in the set discovery algorithm have head specific optimizations.

recover: add a --[no-]verify flag For trivial cases, the cost of the verify run after `hg recover` is getting in the way. In addition for very large repositories, the cost is simply too high to be paid, making `hg recover` an unusable commands. We introduce a --verify flag, set by default. If is automatically associated with a --no-verify flag that one can use to skip the verify step. We might consider changing the default behavior in the future. However this is out of scope for this series.

bundle2: handle compression in _forwardchunks _forwardchunks is used to compensate for getbundle protocol deficits. Since it transparently decodes the payload, it also needs to remove the corresponding compression parameter in case the server decides to send one. This the wire protocol part of issue 5990. Differential Revision: https://phab.mercurial-scm.org/D6182

changelog: parse copy metadata if available in extras This lets read back the copy metadata we just started writing. There are still many places left to teach about getting the copy information from the changeset, but we have enough ({file_copies}, specifically) that we can add it now and have some test coverage of it. Differential Revision: https://phab.mercurial-scm.org/D6186

copies: add config option for writing copy metadata to file and/or changset This introduces a config option that lets you choose to write copy metadata to the changeset extras instead of to filelog. There's also an option to write it to both places. I imagine that may possibly be useful when transitioning an existing repo. The copy metadata is stored as two fields in extras: one for copies since p1 and one for copies since p2. I may need to add more information later in order to make copy tracing faster. Specifically, I'm thinking out recording which files were added or removed so that copies._chaincopies() doesn't have to look at the manifest for that. But that would just be an optimization and that can be added once we know if it's necessary. I have also considered saving space by using replacing the destination file path by an index into the "files" list, but that can also be changed later (but before the feature is ready to release). Differential Revision: https://phab.mercurial-scm.org/D6183

revsetbenchmark: add some simpler revset for heads and roots This revset might leverage compiled code in the future so lets start to track them.

repoview: flag `server.view` as experimental Ideally, the non-experimental version of `experimental.extra-filter-revs` will cover the use case for `server.view=immutable` well enough than having to have this dedicated configuration. Since `server.view` is not part of any release, I would prefer to have it marked as experimental to avoid having it to support it for ever.

repoview: move subsettable in a dedicated module The dictionary got moved in `branchmap` to avoid import cycle. However, we are about to needs it in repoview too. So we introduce a now module to define that that mapping.

upgrade: support upgrade to/from zstd storage (issue6088) Now that we have an official config option for a shiny format improvement, we better make it simple to migrate to/from it.

compression: introduce an official `zstd-revlog` requirement This requirement supersedes `exp-compression-zstd`. However, we keep support for the old requirement. Strictly speaking, we do not need to add a new requirement, there are no logic change making "new" repo incompatible with mercurial client using a version that support `exp-compression-zstd`. The choice to introduce a new requirement is motivated by the following: * The previous requirement was explicitly "experimental". Using it by default could confuse users. * adding support for a hypothetical third compression engine will requires new code, and will comes with its own requirement tool. * We won't use it as the default for a while since I do not think we support zstd on all platform. I can imagine we'll gain another (unrelated but on my default) requirement by the time we turn this zstd by default.

compression: only declare revlog support for available engine Even if we know that an engine support revlog compression, the Mercurial process still won't support it if the compression engine is not available.

branchcache: lazily validate nodes in iteritems() This saves ~0.30 sec on creating a new branch on our internal repo. Differential Revision: https://phab.mercurial-scm.org/D6236

branchcache: only iterate over branches which needs to be verified Otherwise we loop over all the branches and call _verifybranch() even if not required. Differential Revision: https://phab.mercurial-scm.org/D6240

branchcache: fix the docstring of _verifybranch() Initially the function was designed to support verifying all branches but later I decided to have a separate function. I forget to remove the doc related to that. Thanks to Yuya for spotting this in review. Differential Revision: https://phab.mercurial-scm.org/D6239

branchcache: don't verify while creating a copy The copy will not be mark as verified, so there is no need to verify nodes. Thanks to Yuya who spotted this while reviewing. Differential Revision: https://phab.mercurial-scm.org/D6238

pull: improved message issued in case of failed update When running `hg pull --update`, the update step may fail. Nothing in the error message help to understand the abort is related to the secondary step (update) instead of the primary step (pull). We now add some information to the error message to clarify it comes from the update part. It is useful in various situation (uncommitted changes blocking the update, update to hidden destination, etc...) The pull output is updated from: $ hg pull ../repo-Bob --rev 956063ac4557 --update pulling from ../repo-Bob searching for changes adding changesets adding manifests adding file changes added 2 changesets with 0 changes to 2 files (+1 heads) (2 other changesets obsolete on arrival) abort: filtered revision '6'! to: $ hg pull ../repo-Bob --rev 956063ac4557 --update pulling from ../repo-Bob searching for changes adding changesets adding manifests adding file changes added 2 changesets with 0 changes to 2 files (+1 heads) (2 other changesets obsolete on arrival) abort: cannot update to target: filtered revision '6'! (I am not sure why the actual error, "filtered revision '6'", is not using the more modern format mentioning the obsolescence fate of '6')

pull: deal with locally filtered changeset passed into --rev Nowadays, it is possible to explicitly pull a remote revision that end up being hidden locally (eg: obsoleted locally). However before this patch, some internal processing where crashing trying to resolve a filtered revision. Without this patches, the pull output result a confusing output: $ hg pull ../repo-Bob --rev 956063ac4557 pulling from ../repo-Bob searching for changes adding changesets adding manifests adding file changes added 2 changesets with 0 changes to 2 files (+1 heads) (2 other changesets obsolete on arrival) abort: 00changelog.i@956063ac4557828781733b2d5677a351ce856f59: filtered node!

absorb: aborting if another operation is in progress This increases safety of using absorb by both aborting when another operation is in progress (since the absorption could confuse any other command a lot) and holding the locks throughout the reading of the working directory (for which changes to absorb) and the reading of the repo (for which changes to absorb into). Differential Revision: https://phab.mercurial-scm.org/D6237

repoview: keep the branchmap cache for the `served.hidden` view warm For the same reason we want to keep the cache for the `served` view up to date, we want to also keep the `served.hidden` view up to date. If some processes with a readonly access to the repo needs to access it, we better have the cache warm to avoid computing the same data over and over and over. In most case (no secret changesets), the "served" and "served.hidden" set will be identical and no cache will actually have to be updated.

repoview: introduce a filter for serving hidden changesets There are multiple usecase for being able to explicitly view or pull obsolete from a server. We need to be able to do so without exposing the secret changesets. We introduces a dedicated repository "view" to do so. Way to expose this "view" to the user will come later. To keep a behavior consistent with expected client/server behavior, the general idea is for the obsolete access to be explicitly requested by the code generating the request. In addition, the will be server side configuration to restrict the access to this feature.

repoview: fix conditional around unserved changesets The conditional could lead to wrong computation since we have more unserved changesets than just the "secret" phase. In addition, now that we have efficient caching of phased changesets, we don't need the conditional anymore.

branch: return early if we find an open named branch apart from default The current code builds a list of all the open named branches except default and then bool that. This is mostly fine until you get a repo which has thousands of named branches. Differential Revision: https://phab.mercurial-scm.org/D6211

branchcache: don't verify closed nodes in _branchtip() We only do membership testing there. Differential Revision: https://phab.mercurial-scm.org/D6210

branchcache: don't verify closed nodes in iteropen() We expect that the nodes passed to iteropen() will be verified. We are only testing for membership in closed nodes set, so we don't need to verify the whole closed nodes set. This will speed up calculating branchheads() when there are lot of closed nodes related to other branches. Differential Revision: https://phab.mercurial-scm.org/D6209

branchcache: lazily validate nodes from the branchmap On my personal hg-repository with 365 entries in .hg/cache/branch2, following are the numbers for perfbranchmapload. Before this patch: ! wall 0.000866 comb 0.000000 user 0.000000 sys 0.000000 (best of 2680) ! wall 0.001525 comb 0.000000 user 0.000000 sys 0.000000 (max of 2680) ! wall 0.001107 comb 0.001097 user 0.001086 sys 0.000011 (avg of 2680) ! wall 0.001104 comb 0.000000 user 0.000000 sys 0.000000 (median of 2680) With this patch: ! wall 0.000530 comb 0.000000 user 0.000000 sys 0.000000 (best of 4240) ! wall 0.001078 comb 0.000000 user 0.000000 sys 0.000000 (max of 4240) ! wall 0.000696 comb 0.000693 user 0.000677 sys 0.000017 (avg of 4240) ! wall 0.000690 comb 0.000000 user 0.000000 sys 0.000000 (median of 4240) On our internal repository with ~20k entries in branchcache, I see improvement from 0.125 sec to 0.066 sec which is 47% speed up. The above are the numbers of perfbranchmapload which shows how much time we saved by not validating the nodes. But we need to validate some nodes. Following are timings of some mercurial operations which have speed up because of this lazy validation of nodes: No-op `hg update` on our internal repository (Avg on 4 runs): Before: 0.540 secs After: 0.430 secs Setting a branch name which already exists without --force (Avg of 4 runs): Before: 0.510 secs After: 0.250 secs I ran the ASV performance suite and was unable to see any improvements except there was improvement of perfdirstatewrite() on netbeans which I think was not related. I looked into the commit code, the command which I am trying to speedup, it looks like it uses revbranchcache to update the branchcache. Differential Revision: https://phab.mercurial-scm.org/D6208

branchcache: add functions to validate changelog nodes This patch adds functions to validate closed nodes, validate nodes for a certain branch and for all the branches. These functions will be used in upcoming patches. Differential Revision: https://phab.mercurial-scm.org/D6207

readline: provide styled prompt to readline (issue6070) Differential Revision: https://phab.mercurial-scm.org/D6168

copies: move comment about implementation of mergecopies() to end When you start reading about mergecopies(), you want to know what it is, not that there are different implementations depending on config, so this patch moves that comment to the end. By the way, we don't seem to define what "copytracing" is. I'm just leaving it that way because I don't know what it is myself. It seems to be referred to only on mergecopies() (and not in pathcopies(), for example), so maybe "copytracing" is supposed to be exactly what mergecopies() does? Differential Revision: https://phab.mercurial-scm.org/D6235

remotefilelog: return expected type from copies overrides copies._computeforwardmissing() and copies._computenonoverlap() return sets, so the overrides should also do that. Differential Revision: https://phab.mercurial-scm.org/D6234

changelog: extract a _string_unescape() to mirror _string_escape() We use our own _string_escape() to encode the "extras" field. Then we use codecs.escape_decode() to escape it. But there's also a little workaround for dealing with escaped text that looks like octal numbers since the fix for https://bz.mercurial-scm.org/show_bug.cgi?id=3156. This patch extracts the call to codecs.escape_decode() along with the fix for octal numbers and puts it in a _string_unescape(). It also updates the test to check for the octal-number case from the aforementioned bug. As you may have suspected, I want to be able to reuse this new function later. Differential Revision: https://phab.mercurial-scm.org/D6184

copies: extract function for deciding whether to use changeset-centric algos We'll eventually have a "experimental.copies.read-from=changeset-only" option too and I don't want to spread the logic for determining if we should use changeset-centric of filelog-centric algorithms. Differential Revision: https://phab.mercurial-scm.org/D6163

getrenamedfn: get copy data from context object if configured The function returned from getrenamedfn() calls filelog.renamed(). That won't work when storing copy metadata in the changeset. I've just switched to a simple implementation here. We may or may not need to optimize it later, possibly by optimizing the callers. No more tests fail with "--extra-config-opt experimental.copies.read-from=compatibility)" than they did before this patch. Differential Revision: https://phab.mercurial-scm.org/D6162

branchmap: implement __contains__() We have good occurences of `if branch in branchmap()` in our code. If __contains__() is not implemented then it will use __iter__() to find whether the element exists or not which is not good. I am bit confused that whether I should move existing callers to hasbranch() or this patch is a good way. Differential Revision: https://phab.mercurial-scm.org/D6206

branchmap: prevent using __getitem__() in branchheads() branchheads() can directly use self._entries instead. Differential Revision: https://phab.mercurial-scm.org/D6205

branchmap: dynamically resolve type of branchcache class This is required to support subclassing. Thanks to Yuya for suggesting this in D6151. Differential Revision: https://phab.mercurial-scm.org/D6204

copies: print list of divergent renames in sorted order The current order is not obvious to the user (and I may change it in a later patch). Lexicographical order seems like the obvious choice here. Differential Revision: https://phab.mercurial-scm.org/D6227

import: suppress accept: header irccloud and others may perform content negotation and reject (406) if the accept header does not match the mime type of the object

rebase: fix bug that prevented dry-run rebases from printing failures As far as I can tell it should be fine to unconditionally skip _prepareabortorcontinue if we're in the process of raising an Abort here. Differential Revision: https://phab.mercurial-scm.org/D6226

rebase: demonstrate bug in dry-run mode which causes cycles to not be reported Differential Revision: https://phab.mercurial-scm.org/D6225

test: minor cleanup to test-server-view.t While looking into adding error output in this test, I did some cleanup.

repoview: improve documentation for `repo.filtered` method I am sitting next to Joerg Sonnenberger and we are discussion his experience with repoview. This first effect of this discussion is this documentation clarification.

revset: short docstring for checkstatus This is where all the action happens for the status-related revsets, and a little documentation doesn't hurt.

discovery: stop direct use of attribute of partialdiscovery Instead of accessing `undecided` directly for ui display purposes, we introduce a `stats()` method that could be extended in the future with more interesting information. This is in preparation for a forthcoming Rust version of this object. Indeed, attributes and furthermore properties are a bit complicated for classes in native code. We could go further and rename `undecided` to mark it private, but `_undecided` is already taken as support for `_undecided` lazyness.

overlayworkingctx: remove misleading trailing slash from directory pattern The paths passed into the matcher are normalized (this applies to include patterns and regular patterns, and to both glob kind and path kind), so the regex for input "foo/" ended up being "foo(?:/|$)". Once we have a (recursive) pattern kind only for directories, we could switch to that here and remove the "mfiles[0] == path" check. Until then, let's at least make it not misleading. Differential Revision: https://phab.mercurial-scm.org/D6224

overlayworkingctx: fix file/dir audit to be repo-relative Before this patch, test-rebase-inmemory.t would stop erroring out about the conflict if you added a "cd a" before line 252. That was because a glob matcher (which are relative) was unintentionally used. That happened because the matcher was given "include" patterns (not regular patterns), and "include" patterns are always glob by default (i.e. unless you write them including the kind prefix). IOW, the "default='path'" argument passed to ctx.match() was ignored. Differential Revision: https://phab.mercurial-scm.org/D6223

messages: replace some instances of "folder" by "directory" I'm pretty sure this is our preferred term. Differential Revision: https://phab.mercurial-scm.org/D6222

match: fix re2 compability broken in 2e2699af5649 When using re2, we call test_match() instead of match() on the compiled regex object. While match() returns a matcher object or None, test_match() returns True or False. So since 2e2699af5649 running test on a machine with a re2 install fails in many places. Instead we make the code a bit more general and everything goes back to normal.

py3: add b'' prefixes to new doctests in match.py # skip-blame as just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D6221

py3: add one new passing test found by buildbot Differential Revision: https://phab.mercurial-scm.org/D6220

cext: cast s# arguments of Py_BuildValue() to Py_ssize_t The doc doesn't state that "s#" of Py_BuildValue() is controlled by PY_SSIZE_T_CLEAN (unlike the one for PyArg_ParseTuple()), but actually it's switched to Py_ssize_t. https://docs.python.org/2/c-api/arg.html#c.Py_BuildValue https://github.com/python/cpython/blob/2.7/Python/modsupport.c#L432 Follow up for b01bbb8ff1f2 and 896b19d12c08.

remotefilelog: correctly reject wdir filenodes This fixes `hg grep -r 'wdir()'` when remotefilelog is enabled and the working directory contains uncommitted modifications. Differential Revision: https://phab.mercurial-scm.org/D6217

remotefilelog: add tests of `hg grep -r 'wdir()'` This demonstrates how remotefilelog breaks grepping dirtied working directories. A future change will introduce a fix. Differential Revision: https://phab.mercurial-scm.org/D6216

config: read configs from directories in lexicographical order Mercurial currently reads the .rc files specified in HGRCPATH (and the system-default paths) in directory order, which is unspecified. My team at work maintains a set of .rc files. So far there has been no overlap between them, so we had not noticed this behavior. However, we would now like to release some common .rc files and then have another one per plaform with platform-specific overrides. It would be nice if we can determine the load order by choosing names carefully. This patch enables that by loading the .rc files in lexicographical order. Before this patch, the added test case would consistently say "30" on my file system (whatever I have -- some Linux FS). Differential Revision: https://phab.mercurial-scm.org/D6193

remotefilelog: fix crash on `hg addremove` of added-but-deleted file If you `hg add` a file and then delete it from disk, and then run `hg addremove`, the file ends up in the "removed" set that gets passed to the findrenames() override. We then crash because the file is not in the working copy parent. This patch fixes that. Differential Revision: https://phab.mercurial-scm.org/D6194

packaging: ensure that --python is an absolute path when building on Windows For whatever reason, even though only python2 is on PATH, passing `python.exe` causes the later check that it's not py3 to bail out.

packaging: don't crash building wix with python3.6 and earlier `capture_output` was added in 3.7. I was tempted to just check and abort in build.py, since Windows doesn't have the Linux problem where some distros only ship an older python. But this is in a library that could be used elsewhere in the future.

chistedit: add basic colours to diff view This isn't complete, and it would be nice to show the exact same colours that `hg diff` would show. That goal is too lofty, so this just shows some basic colours, on the premise that a little is better than nothing.

chistedit: use default curses colours Terminals will define default colours (for example, white text on black background), but curses doesn't obey those default colours unless told to do so. Calling `curses.use_default_colors` makes curses obey the default terminal colours. One of the most obvious effects is that this allows transparency on terminals that support it. This also brings chistedit closer in appearance to crecord, which also uses default colours. The call may error out if the terminal doesn't support colors, but as far as I can tell, everything still works. If we need a more careful handling of lack of colours, blame me for not doing it now.

match: let regex match function return a boolean Match function for regex pattern kind is built through _buildregexmatch() and _buildmatch() using _rematcher() that returns a re.match function, which either returns a match object or None. This does not conform to Mercurial's matcher interface for __call__() or exact(), which are expected to return a boolean value. We fix this by building a lambda around _rematcher() in _buildregexmatch(). Accordingly, we update doctest examples to remove bool() calls that are now useless.

match: make arguments of _expandsets() optional Arguments 'ctx', 'listsubrepos' and 'badfn' are optional in function body.

match: make _donormalize's auditor and warn arguments optional Argument 'warn' is actually non-required, since there's a 'if warn:' check before usage. Argument 'auditor' is passed to pathutil.canonpath(), in which it is optional.

match: add doctest examples in match() Make the docstring raw, as it now includes escape characters.

match: complete documentation of match() parameters

match: add doctest examples for patkind()

match: add a docstring with doctest examples to patternmatcher Doctest examples aim at illustrating how __call__() and exact() are different, depending on the pattern kind.

match: add doctest examples for exactmatcher Make the docstring raw, since it now includes escape characters.

localrepo: don't allow lookup of working directory revision It seems that repo.lookup(), which is what supports the "lookup" wire protocol command, should not allow the working copy revision input. This fixes both the pull test and the convert test I just added. Differential Revision: https://phab.mercurial-scm.org/D6215

tests: demonstrate broken pull of "ffffffffffff" revision Differential Revision: https://phab.mercurial-scm.org/D6214

tests: demonstrate broken `hg convert` if "ffffffffffff" is in description Differential Revision: https://phab.mercurial-scm.org/D6213

tests: add test of for hash reference translation by `hg convert` The convert extension translates commit references in the commit message. We didn't have any explicit testing of this before, so let's add a test. Differential Revision: https://phab.mercurial-scm.org/D6212

py3: write out hgextindex as bytes in setup.py I hit this trying to build the py2exe target using python3, just to see what would happen. After commenting out `py2exe.Distribution` in setup.py and pointing to a local copy of py2exe that supports python3[1], it complained that `out` was bytes, not str. [1] https://github.com/albertosottile/py2exe/releases/tag/v0.9.3.0

setup: fix a possible NameError on rust build File "setup.py", line 975, in rustbuild "command: %r, environment: %r" % (self.rustsrcdir, cmd, env)) NameError: global name 'cmd' is not defined

crecord: new keys g & G to navigate to the top and bottom respectively This patch introduces two new keys 'g' and 'G' that helps to navigate to the top and bottom of the file/hunk/line respectively. This is inline with the shortcuts used in man, less, more and such tools that makes it convenient to navigate swiftly. 'g' or HOME navigates to the top most file in the ncurses window. 'G' or END navigates to the bottom most file/hunk/line depending on the whether the fold is active or not. If the bottom most file is folded, it navigates to that file and stops there. If the bottom most file is unfolded, it navigates to the bottom most hunk in that file and stops there. If the bottom most hunk is unfolded, it navigates to the bottom most line in that hunk. Differential Revision: https://phab.mercurial-scm.org/D6178

chistedit: properly show verbose diffs I'm not sure if that ever worked and it's an internal API breakage, but `"verbose": True` is not correctly parsed, as most of these options are parsed by diffopts, whereas verbose is a global option. Setting the UI to verbose instead does work and does show a verbose patch, with full commit message. It also shows all files, which unfortunately are a bit hard to read on a single line in the default verbose template. Thus, we also change the default template to use the status template, which shows one file per line as well as its modification state.

interactive: do not prompt about files given in command line For commit and revert commands with --interactive and explicit files given in the command line, we now skip the invite to "examine changes to <file> ? [Ynesfdaq?]". The reason for this is that, if <file> is specified by the user, asking for confirmation is redundant. In patch.filterpatch(), we now use an optional "match" argument to conditionally call the prompt() function when entering a new "header" item. We use .exact() method to compare with files from the "header" in order to only consider (rel)path patterns. Add tests with glob patterns for commit and revert, to make sure we still ask to examine files in these cases.

zstandard: vendor python-zstandard 0.11 The upstream source distribution from PyPI was extracted. Unwanted files were removed. The clang-format ignore list was updated to reflect the new source of files. The project contains a vendored copy of zstandard 1.3.8. The old version was 1.3.6. This should result in some minor performance wins. test-check-py3-compat.t was updated to reflect now-passing tests on Python 3.8. Some HTTP tests were updated to reflect new zstd compression output. # no-check-commit because 3rd party code has different style guidelines Differential Revision: https://phab.mercurial-scm.org/D6199

cext: make osutil.c PY_SSIZE_T_CLEAN This is needed to avoid a deprecation warning on Python 3.8. With this change, we no longer see deprecation warnings for this issue on Python 3.8. Differential Revision: https://phab.mercurial-scm.org/D6198

cext: make parsers.c PY_SSIZE_T_CLEAN This is needed to avoid a deprecation warning in Python 3.8. I believe the conversion of int to Py_ssize_t is harmless in the changed locations. But this being C code, it should be audited with care. Differential Revision: https://phab.mercurial-scm.org/D6197

cext: make revlog.c PY_SSIZE_T_CLEAN Without this, Python 3.8 emits a deprecation warning, as using int for # values is deprecated. Many existing modules use PY_SSIZE_T_CLEAN, so this shouldn't be contentious. I audited the file for all # formatters and verified we are using Py_ssize_t everywhere now. Differential Revision: https://phab.mercurial-scm.org/D6196

tests: add optional output for Python 2.7 deprecation We already had one of these a few lines above. We need it here as well. Differential Revision: https://phab.mercurial-scm.org/D6203

setup: use raw string for regular expression Otherwise Python 3.8 complains about the backslash. Differential Revision: https://phab.mercurial-scm.org/D6202

automation: use raw strings when there are backslashes Otherwise Python 3.8 complains. Differential Revision: https://phab.mercurial-scm.org/D6201

perf: make perf.run-limits code work with Python 3 We need b'' because perf.py isn't run through the source transformer. We need to cast the exception to bytes using pycompat.bytestr() because ValueError can't be %s formatted due to built-in exceptions lacking __bytes__. We need to pycompat.sysstr() before the float() and int() cast so the ValueError message doesn't have b'' in it. Even with that, it looks like the error message for the ValueError for float casts added quotes, so we need to account for that in test output. Differential Revision: https://phab.mercurial-scm.org/D6200

localrepo: rename crev in _filecommit() to cnode, since it's a node I know we often use "rev" generically, but here's it always a node, so it helps to be specific. Differential Revision: https://phab.mercurial-scm.org/D6181

tests: unset environment variable P in test-revset2.t (issue6109) The test tests/test-revset2.t fails the test case "Test repo.anyrevs with customized revset overrides" (line 1609) if the environment variable P is set. The test implicitly expects that the environment, in which it is started, does not export the variable 'P'. To solve this issue, unset 'P' right before the test commands are run. Differential Revision: https://phab.mercurial-scm.org/D6195

hgmanpage: use a py2 and py3 compatible iterable protocol

hgmanpage: use range instead of xrange

packaging: allow to run make with python3 Use "?=", otherwise the variable cannot be set from environment.

cleanup: use set literals where possible Differential Revision: https://phab.mercurial-scm.org/D6192

tests: rename "u" to more usual "ui" in test-context.py Differential Revision: https://phab.mercurial-scm.org/D6191

tests: better document the graft copy case Differential Revision: https://phab.mercurial-scm.org/D6190

py2exe: add workaround to allow bundling of hgext3rd.* extensions py2exe doesn't know how to handle namespace packages *at all*, so it treats them like normal packages. As a result, if we try and bundle hgext3rd.evolve in a py2exe build, it won't work if we install evolve into the virtualenv. In order to work around this, tortoisehg installs hgext3rd.evolve etc into its staged hg directory, since it doesn't use a virtualenv. As a workaround for us, we'll just allow any extra packages users want bundled are part of hg during the pseudo-install phase that py2exe uses. I'm not happy about this, but it *works*. As a sample of how you'd make an MSI with evolve bundled: import os import shutil import subprocess import tempfile def stage_evolve(version): """Stage evolve for inclusion in py2exe binary.""" with tempfile.TemporaryDirectory() as temp: evolve = os.path.join(temp, "evolve") subprocess.check_call([ "hg.exe", "clone", "https://www.mercurial-scm.org/repo/evolve/", "--update", version, evolve, ]) dest = os.path.join('..', 'hgext3rd', 'evolve') if os.path.exists(dest): shutil.rmtree(dest) shutil.copytree(os.path.join(evolve, "hgext3rd", "evolve"), dest) def main(): stage_evolve('tip') print("\0") print("hgext3rd") print("hgext3rd.evolve") print("hgext3rd.evolve.hack") print("hgext3rd.evolve.thirdparty") if __name__ == "__main__": main() is a script you can pass to the wix/build.py as --extra-packages-script, and the resulting .msi will have an hg binary with evolve baked in. users will still need to enable evolve in their hgrc, so you'd probably also want to bundle configs in your msi for an enterprise environment, but that's already easy to do with the support for extra features and wxs files in the wix build process. Differential Revision: https://phab.mercurial-scm.org/D6189

wix: fix the package build when not adding features Should have used ifdef, not if. Sigh. Differential Revision: https://phab.mercurial-scm.org/D6187

histedit: narrow the scope of discarded ui output In 34165875fa5df813bec3a0cd348932b304d44efb, a lot of the output from histedit was excluded. This slightly adjusts the scope of that exclusion, to both discard more uninsteresting messages, and ensure that pre-merge-tool output gets shown before the external merge tool is executed. Differential Revision: https://phab.mercurial-scm.org/D6177

uncommit: abort if an explicitly given file cannot be uncommitted (BC) I've gotten burned several times by this in the last few days. The former tests look simple enough, but if a good file and a bad file are given, the bad files are silently ignored. Some commands like `forget` will warn about bogus files, but that would likely get lost in the noise of an interactive uncommit. The commit command aborts if a bad file is given, so this seems more consistent for commands that alter the repository.

unshelve: disable unshelve during merge (issue5123) As stated in the issue5123, unshelve can destroy the second parent of the context when tried to unshelve with an uncommitted merge. This patch makes unshelve to abort when called with an uncommitted merge. See how shelve.mergefiles works. Commit structure looks like this: ``` ... -> pctx -> tmpwctx -> shelvectx / / second merge parent pctx = parent before merging working context(first merge parent) tmpwctx = commited working directory after merge(with two parents) shelvectx = shelved context ``` shelve.mergefiles first updates to pctx then it reverts shelvectx to pctx with: ``` cmdutil.revert(ui, repo, shelvectx, repo.dirstate.parents(), *pathtofiles(repo, files), **{'no_backup': True}) ``` Reverting tmpwctx files that were merged from second parent to pctx makes them added because they are not in pctx. Changing this revert operation is crucial to restore parents after unshelve. This is a complicated issue as this is not fixing a regression. Thus, for the time being, unshelve during an uncommitted merge can be aborted. (Details taken from http://mercurial.808500.n3.nabble.com/PATCH-V3-shelve-restore-parents-after-unshelve-issue5123-tt4036858.html#a4037408) Differential Revision: https://phab.mercurial-scm.org/D6169

wix: add functionality to inject additional Features into installer This is the last bit required to be able to glue extra configs etc into the installer. Differential Revision: https://phab.mercurial-scm.org/D6180

wix: add support for additional wxs files As with my previous change for an --extra-prebuiild-script, I'm assuming this is predominantly useful in an enterprise environment and am only adding this to wix and not also to inno install scripts. Differential Revision: https://phab.mercurial-scm.org/D6179

wix: add a hook for a prebuild script to inject extra libraries I need this to build packages for Google so we can bundle some extensions in the installed image. My assumption is that this is most interesting for the .msi images so I only wired it up there. I'm not thrilled with the interface this provides, but it was an easy way to retain debug messages on Windows while also having enough structure to know what lines are actually module names for py2exe. Still pending on my end: I need to bundle a couple of config files, and at least one data file. I'm open to advice on how to do those things, and how to do this better. Differential Revision: https://phab.mercurial-scm.org/D6164

compression: introduce an official `format.revlog-compression` option This option supersedes the `experiment.format.compression` option. The value currently supported are zlib (default) and zstd (if Mercurial was compiled with zstd support). The option gained an explicit reference to `revlog` since this is the target usage here. Different storage methods might require different compression strategies. In our tests, using zstd give a significant CPU usage improvement (both compression and decompressing) while keeping similar repository size. Zstd as other interresting mode (dictionnary, pre-text, etc…) that are probably worth exploring. However, just plain switching from zlib to zstd provide a large benefit.

compression: display compression level in debugformat Now that we have options to control the compression level, we teach `hg debugformat` about them. This is a useful information when comparing repositories. Note that we have no trace of the compression level used to store existing deltas. Actually, it would even varies from one delta to another. So we display the currently set value.

compression: introduce a `storage.revlog.zstd.level` configuration This option control the zstd compression level used when compressing revlog chunk. The usage of zstd for revlog compression has not graduated from experimental yet, but we intend to fix that soon. The option name for the compression level is more straight forward to pick, so this changesets comes first. Having a dedicated option for each compression engine is useful because they don't support the same range of values. I ran the same measurement as for the zlib compression level (in the parent changesets). The variation in repository size is stay mostly in the same (small) range. The "read/write" performance see smallish variation, but are overall much better than zlib. Write performance show the same tend of having better write performance for when reaching high-end compression. Again, we don't intend to change the default zstd compression level (currently: 3) in this series. However this is worth investigating in the future. The Performance comparison of zlib vs zstd is quite impressive. The repository size stay in the same range, but the performance are much better in all situations. Comparison summary ================== We are looking at: - performance range for zlib - performance range for zstd - comparison of default zstd (level-3) to default zlib (level 6) - comparison of the slowest zstd time to the fastest zlib time Read performance: ----------------- | zlib | zstd | cmp | f2s mercurial | 0.170159 - 0.189219 | 0.144127 - 0.149624 | 80% | 88% pypy | 2.679217 - 2.768691 | 1.532317 - 1.705044 | 60% | 63% netbeans | 122.477027 - 141.620281 | 72.996346 - 89.731560 | 58% | 73% mozilla | 147.867662 - 170.572118 | 91.700995 - 105.853099 | 56% | 71% Write performance: ------------------ | zlib | zstd | cmp | f2s mercurial | 53.250304 - 56.2936129 | 40.877025 - 45.677286 | 75% | 86% pypy | 460.721984 - 476.589918 | 270.545409 - 301.002219 | 63% | 65% netbeans | 520.560316 - 715.930400 | 370.356311 - 428.329652 | 55% | 82% mozilla | 739.803002 - 987.056093 | 505.152906 - 591.930683 | 57% | 80% Raw data -------- repo alg lvl .hg/store size 00manifest.d read write mercurial zlib 1 49,402,813 5,963,475 0.170159 53.250304 mercurial zlib 6 47,197,397 5,875,730 0.182820 56.264320 mercurial zlib 9 47,121,596 5,849,781 0.189219 56.293612 mercurial zstd 1 49,737,084 5,966,355 0.144127 40.877025 mercurial zstd 3 48,961,867 5,895,208 0.146376 42.268142 mercurial zstd 5 48,200,592 5,938,676 0.149624 43.162875 mercurial zstd 10 47,833,520 5,913,353 0.145185 44.012489 mercurial zstd 15 47,314,604 5,728,679 0.147686 45.677286 mercurial zstd 20 47,330,502 5,830,539 0.145789 45.025407 mercurial zstd 22 47,330,076 5,830,539 0.143996 44.690460 pypy zlib 1 370,830,572 28,462,425 2.679217 460.721984 pypy zlib 6 340,112,317 27,648,747 2.768691 467.537158 pypy zlib 9 338,360,736 27,639,003 2.763495 476.589918 pypy zstd 1 362,377,479 27,916,214 1.532317 270.545409 pypy zstd 3 354,137,693 27,905,988 1.686718 294.951509 pypy zstd 5 342,640,043 27,655,774 1.705044 301.002219 pypy zstd 10 334,224,327 27,164,493 1.567287 285.186239 pypy zstd 15 329,000,363 26,645,965 1.637729 299.561332 pypy zstd 20 324,534,039 26,199,547 1.526813 302.149827 pypy zstd 22 324,530,595 26,198,932 1.525718 307.821218 netbeans zlib 1 1,281,847,810 165,495,457 122.477027 520.560316 netbeans zlib 6 1,205,284,353 159,161,207 139.876147 715.930400 netbeans zlib 9 1,197,135,671 155,034,586 141.620281 678.297064 netbeans zstd 1 1,259,581,737 160,840,613 72.996346 370.356311 netbeans zstd 3 1,232,978,122 157,691,551 81.622317 396.733087 netbeans zstd 5 1,208,034,075 160,246,880 83.080549 364.342626 netbeans zstd 10 1,188,624,176 156,083,417 79.323935 403.594602 netbeans zstd 15 1,176,973,589 153,859,477 89.731560 428.329652 netbeans zstd 20 1,162,958,258 151,147,535 82.842667 392.335349 netbeans zstd 22 1,162,707,029 151,150,220 82.565695 402.840655 mozilla zlib 1 2,775,497,186 298,527,987 147.867662 751.263721 mozilla zlib 6 2,596,856,420 286,597,671 170.572118 987.056093 mozilla zlib 9 2,587,542,494 287,018,264 163.622338 739.803002 mozilla zstd 1 2,723,159,348 286,617,532 91.700995 570.042751 mozilla zstd 3 2,665,055,001 286,152,013 95.240155 561.412805 mozilla zstd 5 2,607,819,817 288,060,030 101.978048 505.152906 mozilla zstd 10 2,558,761,085 283,967,648 104.113481 497.771202 mozilla zstd 15 2,526,216,060 275,581,300 105.853099 591.930683 mozilla zstd 20 2,485,114,806 266,478,859 95.268795 576.515389 mozilla zstd 22 2,484,869,080 266,456,505 94.429282 572.785537

compression: introduce a `storage.revlog.zlib.level` configuration This option control the zlib compression level used when compression revlog chunk. This is also a good excuse to pave the way for a similar configuration option for the zstd compression engine. Having a dedicated option for each compression algorithm is useful because they don't support the same range of values. Using a higher zlib compression impact CPU consumption at compression time, but does not directly affected decompression time. However dealing with small compressed chunk can directly help decompression and indirectly help other revlog logic. I ran some basic test on repositories using different level. I am using the mercurial, pypy, netbeans and mozilla-central clone from our benchmark suite. All tested repository use sparse-revlog and got all their delta recomputed. The different compression level has a small effect on the repository size (about 10% variation in the total range). My quick analysis is that revlog mostly store small delta, that are not affected by the compression level much. So the variation probably mostly comes from better compression of the snapshots revisions, and snapshot revision only represent a small portion of the repository content. I also made some basic timings measurements. The "read" timings are gathered using simple run of `hg perfrevlogrevisions`, the "write" timings using `hg perfrevlogwrite` (restricted to the last 5000 revisions for netbeans and mozilla central). The timings are gathered on a generic machine, (not one of our performance locked machine), so small variation might not be meaningful. However large trend remains relevant. Keep in mind that these numbers are not pure compression/decompression time. They also involve the full revlog logic. In particular the difference in chunk size has an impact on the delta chain structure, affecting performance when writing or reading them. On read/write performance, the compression level has a bigger impact. Counter-intuitively, the higher compression levels improve "write" performance for the large repositories in our tested setting. Maybe because the last 5000 delta chain end up having a very different shape in this specific spot? Or maybe because of a more general trend of better delta chains thanks to the smaller chunk and snapshot. This series does not intend to change the default compression level. However, these result call for a deeper analysis of this performance difference in the future. Full data ========= repo level .hg/store size 00manifest.d read write ---------------------------------------------------------------- mercurial 1 49,402,813 5,963,475 0.170159 53.250304 mercurial 6 47,197,397 5,875,730 0.182820 56.264320 mercurial 9 47,121,596 5,849,781 0.189219 56.293612 pypy 1 370,830,572 28,462,425 2.679217 460.721984 pypy 6 340,112,317 27,648,747 2.768691 467.537158 pypy 9 338,360,736 27,639,003 2.763495 476.589918 netbeans 1 1,281,847,810 165,495,457 122.477027 520.560316 netbeans 6 1,205,284,353 159,161,207 139.876147 715.930400 netbeans 9 1,197,135,671 155,034,586 141.620281 678.297064 mozilla 1 2,775,497,186 298,527,987 147.867662 751.263721 mozilla 6 2,596,856,420 286,597,671 170.572118 987.056093 mozilla 9 2,587,542,494 287,018,264 163.622338 739.803002

compression: accept level management for zlib compression We update the zlib related class to be support setting the compression level. This changeset focus on updating the internal only. A way to configure this level will be introduced in the next changeset.

util: extract compression code in `mercurial.utils.compression` The code seems large enough to be worth extracting. This is similar to what was done for various module in `mercurial/utils/`. Since None of the compression logic takes a `ui` objet, issuing deprecation warning is tricky. Luckly the logic does not seems to have many external users.

merge: make "labels" argument to graft() optional, like it is for update() graft() just passes the argument on to update(), and update() doesn't require it, so graft() shouldn't either. Differential Revision: https://phab.mercurial-scm.org/D6175

revset: remove comment about linkrev workaround from user-facing docs I think the code is clear enough so we don't need to keep the comment at all (by now, most Mercurial developers are probably familiar with the linkrevs issues). Differential Revision: https://phab.mercurial-scm.org/D6176

shelve: let cmdutil.revert() take care of backing up untracked files cmdutil.revert() backs up untracked files, so I don't see a reason to do it shelve.mergefiles(). We have tests for this and they still pass. Differential Revision: https://phab.mercurial-scm.org/D6174

shelve: stop passing list of files to revert It seems to work just fine to not specify any files here. I suspect it looked the way it did for historical reasons. It apparently used to use merge instead of rebase until 1d7a36ff2615 (shelve: use rebase instead of merge (issue4068), 2013-10-23) and it makes sense to want to restrict the set of files then. I noticed this because of the files.extend(shelvectx.p1().files()). If the working copy was clean before, then shelvectx.p1() will be the working copy parent and that ended up adding all the files in that set. In our Google-internal Mercurial setup (including a FUSE) that was very noticeably slow when the working copy parent happened to have many files in large directories. This patch doesn't yet remove the call to shelvectx.p1().files(). We also use that set for deciding what to back up. I'm pretty sure it's safe to back up only the set of files we already back even if we no longer restrict the set of files to revert, so this patch should be safe on its own. Regardless, the next patch will delegate the backing-up to cmdutil.revert(). Incidentally, this also gets rid of a repo.pathto() that I had earlier wanted to get rid of. Differential Revision: https://phab.mercurial-scm.org/D6173

remotefilelog: prefetch files in deterministic order I have been troubleshooting some slowness in this area (it's unclear if it's the client or server that's to blame, but that's beside the point) and it's a lot easier to do troubleshoot if the files are prefetched in the same order each time. Differential Revision: https://phab.mercurial-scm.org/D6172

debugdiscovery: display time elapsed during the discovery step This is a useful information. Now that we perform more analysing after the discovery is done, it is worth have a more precise measurement. For serious timing analysis use `hg perfdiscovery`.

debugdiscovery: only list common heads on verbose The list of common heads is only part of the useful information. In addition on repository with many heads, the information is very not helpful (just fill a couple of screen with hash). As a result we hide it behind a --verbose flag.

debugdiscovery: drop duplicated information The old line informing about the local being a superset or subset of the remote is redundant with the newly introduced data. So we drop it.