context: rename troubles into instabilities Rename troubles context method into instabilities. Copy the old troubles method and add a deprecation warning. This way extensions calling troubles will see the deprecation warning but will not break due to new return values. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D238

repair: move manifest strip to a separate function This moves manifest stripping to a separate function so implementations of the manifest that don't support stripping can replace this function with a no-op. I considered adding a strip api to the manifestlog, so other implementations could make it a no-op there, but it seems like strip might be unique to the revlog implementation, and therefore shouldn't be present on the generic api. Differential Revision: https://phab.mercurial-scm.org/D292

repair: refactor broken linkrev collection This refactors broken linkrev collection such that manifest collection is in a separate function. This allows extensions to replace the manifest collection with a non-revlog oriented version. I considered moving the collect changes function onto the manifestlog itself, so it would be behind the abstraction, but since the store we're building doesn't even have the concept of strip, embeding that concept in the manifestlog api seemed odd. Differential Revision: https://phab.mercurial-scm.org/D291

tests: update duplicateoptions test to use bytestrings everywhere Test now passes on Python 3. Differential Revision: https://phab.mercurial-scm.org/D297

extensions: attempt to use non-deprecated inspect method Avoids some deprecation warnings when extension loading breaks. Differential Revision: https://phab.mercurial-scm.org/D295

tests: clean up many print statements to be print functions instead Differential Revision: https://phab.mercurial-scm.org/D294

tests: fix simple heredoc print statements to work on Py3 There are a handful left that require some more care. Differential Revision: https://phab.mercurial-scm.org/D293

obsolete: use bytes() instead of str() so the node is bytes on py3 I'm not sure this is right, since this should either be bytes or str to match what's going on in the revlog layer. Differential Revision: https://phab.mercurial-scm.org/D271

tests: fix test-notify.t to use $PYTHON In doing so, I consolidated the filtering logic. This resulted in some small test output changes, but I think the consistency throughout the test is worth it. Differential Revision: https://phab.mercurial-scm.org/D279

tests: make test-highlight code portable to python3 This is easier than trying to do some sort of check-code shenanigans. Differential Revision: https://phab.mercurial-scm.org/D278

dagparser: make print statement in doctest Py3 portable Differential Revision: https://phab.mercurial-scm.org/D277

i18n: fix check-translation.py to be less broken on Python 3 These are all simple one-argument print statements, so this syntax works the same way in 2 and 3. Differential Revision: https://phab.mercurial-scm.org/D276

ui: refactor extractchoices so it doesn't break on Python 3 Differential Revision: https://phab.mercurial-scm.org/D275

obsutil: defend against succsmarkers() returning None I'm not sure if this is a realistic problem, but doing this avoids some pretty awful test failures on Python 3, and it looks like it should be harmless. Differential Revision: https://phab.mercurial-scm.org/D274

changegroup: more **kwargs Differential Revision: https://phab.mercurial-scm.org/D273

bundle2: convert ints to strings using pycompat.bytestring() Fixes some Python 3 regressions. We don't use %d here because the part id is actually an Optional[int]. It should always be initialized to a non-None value by the time this code executes, but we shouldn't blindly depend on that being the case. Differential Revision: https://phab.mercurial-scm.org/D272

tests: fix up test-run-tests failures on Python 3.6 Some of the recent work on run-tests.py didn't work on Python 3. This fixes the regressions. Differential Revision: https://phab.mercurial-scm.org/D270

template: rename troubles templatekw into instabilities Rename troubles template keyword into instabilities and add a deprecation warning on templatekw. Update default mapfile and test files to use the new template keyword. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D237

test-rebase: add a brute force test Rebase is becoming more complex and it looks like a good idea to try some brute force enumeration to cover cases that are hard to find manually. Using brute force to generate repos in different shapes and enumerating the rebase source and destination would generate too many cases that takes too long to compute. This patch limits the "brute force" to only the "rebase source" part. Repo and destination are still manual. The added test cases are crafted manually to reveal some behaviors that are not covered by other tests: - "revlog index out of range" crash - after rebase, p1 == p2, p2 != null - "nothing to merge" abort In the future we might want to add more tests here. For now I'm more interested in revealing interesting behaviors in a minified way. I tried some more complex cases but didn't find other interesting behaviors. Differential Revision: https://phab.mercurial-scm.org/D262

dirstate: simplify dirstate's __iter__ Probably also a performance win, but not measurable in perfdirstate. Differential Revision: https://phab.mercurial-scm.org/D269

repo: skip invalidation of changelog if it has 'delayed' changes (API) The changelog object can store recently added revisions in memory until the transaction is committed. We don't want to lose those changes even if repo.invalidate(clearfilecache=True), so let's skip the changelog when it has such 'delayed' changes. Differential Revision: https://phab.mercurial-scm.org/D152

httppeer: make several instance attributes internal (API) Peer instances are supposed to conform to a well-defined API so consumers can be agnostic about the underlying peer type. To reinforce this, this commit renames a handful of instance attributes on httpeer so they no longer have a "public" name. Differential Revision: https://phab.mercurial-scm.org/D268

httppeer: remove unused handler attribute The consumer of this attribute was removed by b6bd4ee6ed85 in 2008. Differential Revision: https://phab.mercurial-scm.org/D267

localrepo: remove unused requirements attributes on localpeer (API) The previous changeset removed the last consumer of requirements. I'm not sure when supportedformats became unused. But I couldn't find any obvious instances where it is being used. It likely stems from peers being derived from repository instances several years ago and is a holdover from that day. Differential Revision: https://phab.mercurial-scm.org/D266

exchange: access requirements on repo instead of peer As part of formalizing the peer interface, I audited for attribute accesses for non-internal names to find API violations. This uncovered the code changed in this commit. localpeer.requirements is just an alias to the repo's requirements attribute. So, change the code to get the data from the source instead of relying on a one-off attribute in the localpeer type. Differential Revision: https://phab.mercurial-scm.org/D265

exchange: drop support for lock-based unbundling (BC) Locking over the wire protocol and the "addchangegroup" wire protocol command has been deprecated since e8c4f3d3df8c, which was first part of Mercurial 0.9.1. Support for handling these commands from sshserver was dropped in 9f6e0e7ef828 in 2015, effectively locking out pre 0.9.1 clients from new servers. However, client-side code for calling lock and addchangegroup is still present in exchange.py and the various peer classes to facilitate pushing to pre 0.9.1 servers. The lock-based pushing mechanism is extremely brittle. 0.9.1 was released in July 2006 and I highly doubt anyone is still running such an ancient version of Mercurial on a server. I'm about to refactor the peer API and I don't think it is worth keeping support for this ancient protocol feature. So, this commit removes client support for the lock-based pushing mechanism. This means modern clients will no longer be able to push to pre 0.9.1 servers. Differential Revision: https://phab.mercurial-scm.org/D264

http: add a test of actually pushing with httppostargs This was previously untested. Sigh. Differential Revision: https://phab.mercurial-scm.org/D230

filemerge: move decorator definition for internal merge tools to registrar This patch also adds extra loading entry for internal merge tools to extensions.py, for similarity to other decorators defined in registrar.py. This patch uses "internalmerge" for decorator class name, instead of original "internaltool", because the latter is too generic. BTW, after this patch, 4-spaces indentation is added to the 1st line of internal merge tool description docstring, and this may make already translated entries in *.po fuzzy. Even though this indentation is required for "definition list" in reST syntax, absence of it has been overlooked, because help.makeitemsdoc() forcibly inserts it at generation of online help. But this forcible insertion causes formatting issue (I'll send another patch series for this). Therefore, this additional indentation should be reasonable.

releasenotes: add similarity check function to compare incoming notes It is possible that the incoming note fragments have some similar content as the existing release notes. In case of a bug fix, we match for issueNNNN in the existing notes. For other general cases, it makes use of fuzzywuzzy library to get a similarity score. If the score is above a certain threshold, we ignore the fragment, otherwise add it. But the score might be misleading for small commit messages. So, it uses similarity function only if the length of string (in words) is above a certain value. The patch adds tests related to its usage. But it needs improvement in the sense of combining incoming notes. We can use interactive mode for adding notes. Maybe we can do this if similarity is under a certain range.

releasenotes: add import check for fuzzywuzzy This patch adds the has_fuzzywuzzy for import check of external dependency fuzzywuzzy.

run-tests: don't drop optional lines after a missing unconditional line The previous behavior was to bail out when hitting 'awol', so 'missing (?)' was also dropped.

run-tests: drop required (feature !) style lines when the output is missing Essentially, these were acting as a verbose (?) flag, since they weren't being dropped when required. Foozy has a nice description [1]. Basically, a couple more places needed to check the features before treating it as optional. I don't like how test-run-tests.py had to be hacked, but _hghave() can't be made a static method. The test change was a change while developing `debugssl`, prior to tightening up the cases where the message is printed, that this fix would have caught. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-July/101941.html

status: avoid recursing into ignored directory with "--terse u" Let "isignoreddir" function first check that supplied directory is itself ignored before walking recursively into its content. Otherwise, the command is awfully slow when one has an ignored directory with a lot of content. Update and rephrase function docstring accordingly.

pushvars: move fb extension pushvars to core pushvars extension in fbext adds a --pushvars flag to push command using which one send strings to server which becomes environment variables there prepended with HG_USERVAR_. These variables can then be used to run hooks on the server. The extension is moved directly to core and unbundling of the strings and converting them to environment variables at server is disabled by default for security reasons. One can turn that on by following config: [push] pushvars.server = true This patch also adds the test for the extension. Differential Revision: https://phab.mercurial-scm.org/D210

phabricator: update diff property even if we choose not to create a new diff The diff property contains metadata like "HG Node". Previously we skip uploading a new diff if we are sure that the old patch and new patch have a same content. That has issues when a pusher adds an obsmarker using the old "HG Node" stored in the old diff. This patch adds logic to update the diff property so "HG Node" gets updated to prevent that issue. Differential Revision: https://phab.mercurial-scm.org/D229

phabricator: use Phabricator's last node information This makes it more strict when checking whether or not we should update a Differential Revision. For example, a) Alice updates D1 to content 1. b) Bob updates D1 to content 2. c) Alice tries to update D1 to content 1. Previously, `c)` will do nothing because `phabsend` detects the patch is not changed. A more correct behavior is to override Bob's update here, hence the patch. This also makes it possible to return a reaonsable "last node" when there is no tags but only `Differential Revision` commit messages. Test Plan: ``` for i in A B C; do echo $i > $i; hg ci -m $i -A $i; done hg phabsend 0:: # D40: created # D41: created # D42: created echo 3 >> C; hg amend; hg phabsend . # D42: updated hg tag --local --hidden -r 2 -f D42 # move tag to the previous version hg phabsend . # D42: skipped (previously it would be "updated") rm -rf .hg; hg init hg phabread --stack D42 | hg import - hg phabsend . # D42: updated hg tag --local --remove D42 hg commit --amend hg phabsend . # D42: updated (no new diff uploaded, previously it will upload a new diff) ``` The old diff object is now returned, which could be useful in the next patch. Differential Revision: https://phab.mercurial-scm.org/D121

phabricator: add --confirm option to phabsend command This adds a --confirm flag similar to the confirm flag of `hg email` using which one can confirm the changesets before they get emailed. The confirm flag will show the changesets and ask for confirmation before sending them. Differential Revision: https://phab.mercurial-scm.org/D218

evolution: rename bumped to phase-divergent Rename bumped to phase-divergent in all external user-facing output. Only update user-facing output for the moment, variables names, templates keyword and potentially configuration would be done in later series. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D216

evolution: rename divergent to content-divergent Rename divergent to content-divergent in all external user-facing output. Only update user-facing output for the moment, variables names, templates keyword and potentially configuration would be done in later series. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D215

py3: use bytes IO to write sample hgrc Unicode sucks. Stop using Text IO and manually convert line endings.

buildrpm: do not break in presence of custom user configs For example, if "hg log" was defined as an alias: # /etc/mercurial/hgrc [alias] log = log --graph the buildrpm script would be surprised by log messages formatted in unexpected ways, and bail out. This patch sets HGPLAIN, effectively resetting all the user configs, including log output, to a common state, making the build more predictable across all the possible environments.

sparse: treat paths as cwd-relative This commit makes it so sparse treats passed paths as CWD-relative, not repo-root-realive. This is a more intuitive behavior in my (and some other FB people's) opinion. This is breaking change however. My hope here is that since sparse is experimental, it's ok to introduce BCs. The reason (glob)s are needed in the test is this: in these two cases we do not supply path together with slashes, but `os.path.join` adds them, which means that under Windows they can be backslashes. To demonstrate this behavior, one could remove the (glob)s and run `./run-tests.py test-sparse.t` from MinGW's terminal on Windows.

match: expose some data and functionality to other modules This patch makes sure that other modules can check whether patterns are CWD-relative.

sparse: properly error out when absolute paths are used Current logic is misleading (it says it drops only absolute paths, but it actually drops all of them), not cross-platform (does not support Windows) and IMO just wrong (as it should just error out if absolute paths are given). This commit fixes it.

py3: convert arbitrary exception object to byte string more reliably Our exception types implement __bytes__(), which should be tried first. Do lossy encoding conversion as a last resort.

build: delay version computation on macOS builds The way HGVER is evaluated now, it'll be evaluated at the beginning of the make execution - with this change, it's evaluated when it gets to that command, at which point the version file it's looking for is sure to exist and be up-to-date. Differential Revision: https://phab.mercurial-scm.org/D224

color: remove warnings if term is not formatted (==dumb or !ui.formatted()) If the user sets color.mode=terminfo, and then runs in the shell inside of emacs (so TERM=dumb), the previous behavior was that it would warn about no terminfo entry for setab/setaf, and then warn about 'failed to set color mode to terminfo'. The first warning is silenced by carrying 'formatted' through to _terminfosetup, the second is silenced by using 'formatted' instead of ui.formatted(). If --color=on (or ui.color=always) is specified, this will still warn, since the formatted boolean is set to true in these cases. Differential Revision: https://phab.mercurial-scm.org/D223

rpms: add chg I'm not sure if there's a reason chg is not added by default. If not, I would like to propose adding in this patch. Differential Revision: https://phab.mercurial-scm.org/D220

tests: demonstrate crash when trying to rebase merge without its parents As the test case shows, when "hg rebase -d G -r 'B + D + F'" is run on the following graph, we crash with traceback. It's reasonable to fail because we can not easily produce a correct rebased F. The problem is what diff to apply to either the rebased B or the rebased D. We could potentially produce the result by e.g. applying the (F-D) diff to the rebased B and then applying the reverse (E-D) diff on top, but that could result in merge conflicts in each of those steps, which we don't have a way of dealing with. So for now, let's just add a test case to demonstrate that we crash (i.e. the AssertionError is clearly incorrect since the user can run into it). F /| C E | | B D G \|/ A Differential Revision: https://phab.mercurial-scm.org/D212

tests: demonstrate broken rebase of merge with p1's successor in dest The fix in 8ede973597fd (rebase: handle successor targets (issue5198), 2016-04-11) only fixed the case where p2's successor was in the destination, and only when the successor was exactly the destination (i.e. not when the successor was an ancestor of it). This patch adds a test case for when p1's successor is in the destination. It adds another one for when the successor is an ancestor of the destination. To do that simply, it also rewrites the test case using drawdag. Differential Revision: https://phab.mercurial-scm.org/D211

bundle2: obtain repr() of exception in a python3-safe way This was exposed by other problems in bundle generation, but I'm not sure how to test it for now.

bundle2: use bytestr() instead of str() to convert part id to bytes This was exposed by trying to run previously-passing Python 3 tests.

bundle2: work around zip() being lazy in Python 3

bundle2: look for __next__ as well as next to identify iterators In Python 3, next is called __next__ and this was failing to catch some iterators.

bundle2: use modern Python division This was failing on Python 3 because the / was returning a float, which was then making the __mul__ on a bytes sad.

changegroup: wrap some ** expansions in strkwargs

obsolete: reuse _fm1metapair for computing _fm1metapairsize It's evaluated at import time, so it seems silly to not reuse the constant name.

setup: fix installing in a mingw environment The addition, in 9a4adc76c88a, of a hack for the MSVC compiler class was overwriting the original class for the Mingw32CCompiler class, leading to an error when the HackedMingw32CCompiler is instantiated. Differential Revision: https://phab.mercurial-scm.org/D329

osx: delay version computation on macOS builds Before this patch, HGVER would be evaluated at the beginning of the make execution, and would be unset because build/mercurial/ doesn't exist yet at that point. Now we compute the version after the `make install` run has completed. This is backported to stable from 8626b44516c1, but that revision had an error in the shell invocation syntax.

evolution: rename unstable to orphan Rename unstable to orphan in all external user-facing output. Only update user-facing output for the moment, variables names, templates keyword and potentially configuration would be done in later series. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D214

evolution: rename trouble(s) to instability Rename trouble(s) to instability in all external user-facing output. Only update user-facing output for the moment, variables names, templates keyword and potentially configuration would be done in later series. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D213

bundle2: load hookargs from bundleoperation into transaction when started one When a transaction is started, we must load the hookargs from the bundleoperation object to the transaction so that they can be used in the transaction. Also this patch makes sure no more hookargs are added to the bundleoperation object once the transaction starts. This is a part of porting fb extension bundle2hooks to core. Differential Revision: https://phab.mercurial-scm.org/D209

bundle2: add the capability to store hookargs on bundle operation object There are extensions like pushrebase, pushvars which run hooks on a server before taking the lock. Since the lock is not taken, transaction is not there, so the hookargs can't be stored on the transaction. Adding hooksargs to bundle operation object will help in running hooks before taking the lock. This is a part of moving fb's extension bundle2hooks to core. Differential Revision: https://phab.mercurial-scm.org/D208

branchmap: revert c34532365b38 for Python 2.7 compatibility Old versions of python 2.7 don't like that the second argument to struct.unpack_from is a bytearray, so the change removing the util.buffer around that argument in branchmap broke running on older versions of python 2.7. Differential Revision: https://phab.mercurial-scm.org/D330

Added signature for changeset 3fee7f7d2da0

Added tag 4.3.1 for changeset 3fee7f7d2da0

ssh: unban the use of pipe character in user@host:port string This vulnerability was fixed by the previous patch and there were more ways to exploit than using '|shellcmd'. So it doesn't make sense to reject only pipe character. Test cases are updated to actually try to exploit the bug. As the SSH bridge of git/svn subrepos are not managed by our code, the tests for non-hg subrepos are just removed. This may be folded into the original patches.

ssh: quote parameters using shellquote (SEC) This patch uses shellquote to quote ssh parameters more strictly to avoid shell injection.

subrepo: add tests for git rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

subrepo: add tests for svn rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

subrepo: add tests for hg rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

push: add tests for unsafe ssh url (SEC)

pull: add tests for unsafe ssh url (SEC)

clone: add tests for unsafe ssh url (SEC)

sshpeer: check for safe ssh url (SEC) Checking in the sshpeer for a rogue ssh:// urls seems like the right place to do it (instead of whack-a-mole with pull, clone, push, etc).

ssh: ban any username@host or host that starts with - (SEC) This paranoia probably isn't required, but it can't hurt either.

util: add utility method to check for bad ssh urls (SEC) Our use of SSH has an exploit that will parse the first part of an url blindly as a hostname. Prior to this set of security patches, a url with '-oProxyCommand' could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' can be abused to execute arbitrary commands in a similar fashion. We defend against this by checking ssh:// URLs and looking for a hostname that starts with a - or contains a |. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

pathauditor: disable cache of audited paths by default (issue5628) The initial attempt was to discard cache when appropriate, but it appears to be error prone. We had to carefully inspect all places where audit() is called e.g. without actually updating filesystem, before removing files and directories, etc. So, this patch disables the cache of audited paths by default, and enables it only for the following cases: - short-lived auditor objects - repo.vfs, repo.svfs, and repo.cachevfs, which are managed directories and considered sort of append-only (a file/directory would never be replaced with a symlink) There would be more cacheable vfs objects (e.g. mq.queue.opener), but I decided not to inspect all of them in this patch. We can make them cached later. Benchmark result: - using old clone of http://selenic.com/repo/linux-2.6/ (38319 files) - on tmpfs - run HGRCPATH=/dev/null hg up -q --time tip && hg up -q null - try 4 times and take the last three results original: real 7.480 secs (user 1.140+22.760 sys 0.150+1.690) real 8.010 secs (user 1.070+22.280 sys 0.170+2.120) real 7.470 secs (user 1.120+22.390 sys 0.120+1.910) clearcache (the other series): real 7.680 secs (user 1.120+23.420 sys 0.140+1.970) real 7.670 secs (user 1.110+23.620 sys 0.130+1.810) real 7.740 secs (user 1.090+23.510 sys 0.160+1.940) enable cache only for vfs and svfs (this series): real 8.730 secs (user 1.500+25.190 sys 0.260+2.260) real 8.750 secs (user 1.490+25.170 sys 0.250+2.340) real 9.010 secs (user 1.680+25.340 sys 0.280+2.540) remove cache function at all (for reference): real 9.620 secs (user 1.440+27.120 sys 0.250+2.980) real 9.420 secs (user 1.400+26.940 sys 0.320+3.130) real 9.760 secs (user 1.530+27.270 sys 0.250+2.970)

tests: show cache of audited paths is never invalidated

stable: merge heads

Added signature for changeset 943c91326b23

Added tag 4.2.3 for changeset 943c91326b23

ssh: unban the use of pipe character in user@host:port string This vulnerability was fixed by the previous patch and there were more ways to exploit than using '|shellcmd'. So it doesn't make sense to reject only pipe character. Test cases are updated to actually try to exploit the bug. As the SSH bridge of git/svn subrepos are not managed by our code, the tests for non-hg subrepos are just removed. This may be folded into the original patches.

ssh: quote parameters using shellquote (SEC) This patch uses shellquote to quote ssh parameters more strictly to avoid shell injection.

subrepo: add tests for git rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

subrepo: add tests for svn rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

subrepo: add tests for hg rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

push: add tests for unsafe ssh url (SEC)

pull: add tests for unsafe ssh url (SEC)

clone: add tests for unsafe ssh url (SEC)

sshpeer: check for safe ssh url (SEC) Checking in the sshpeer for a rogue ssh:// urls seems like the right place to do it (instead of whack-a-mole with pull, clone, push, etc).

ssh: ban any username@host or host that starts with - (SEC) This paranoia probably isn't required, but it can't hurt either.

util: add utility method to check for bad ssh urls (SEC) Our use of SSH has an exploit that will parse the first part of an url blindly as a hostname. Prior to this set of security patches, a url with '-oProxyCommand' could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' can be abused to execute arbitrary commands in a similar fashion. We defend against this by checking ssh:// URLs and looking for a hostname that starts with a - or contains a |. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

pathauditor: disable cache of audited paths by default (issue5628) The initial attempt was to discard cache when appropriate, but it appears to be error prone. We had to carefully inspect all places where audit() is called e.g. without actually updating filesystem, before removing files and directories, etc. So, this patch disables the cache of audited paths by default, and enables it only for the following cases: - short-lived auditor objects - repo.vfs, repo.svfs, and repo.cachevfs, which are managed directories and considered sort of append-only (a file/directory would never be replaced with a symlink) There would be more cacheable vfs objects (e.g. mq.queue.opener), but I decided not to inspect all of them in this patch. We can make them cached later. Benchmark result: - using old clone of http://selenic.com/repo/linux-2.6/ (38319 files) - on tmpfs - run HGRCPATH=/dev/null hg up -q --time tip && hg up -q null - try 4 times and take the last three results original: real 7.480 secs (user 1.140+22.760 sys 0.150+1.690) real 8.010 secs (user 1.070+22.280 sys 0.170+2.120) real 7.470 secs (user 1.120+22.390 sys 0.120+1.910) clearcache (the other series): real 7.680 secs (user 1.120+23.420 sys 0.140+1.970) real 7.670 secs (user 1.110+23.620 sys 0.130+1.810) real 7.740 secs (user 1.090+23.510 sys 0.160+1.940) enable cache only for vfs and svfs (this series): real 8.730 secs (user 1.500+25.190 sys 0.260+2.260) real 8.750 secs (user 1.490+25.170 sys 0.250+2.340) real 9.010 secs (user 1.680+25.340 sys 0.280+2.540) remove cache function at all (for reference): real 9.620 secs (user 1.440+27.120 sys 0.250+2.980) real 9.420 secs (user 1.400+26.940 sys 0.320+3.130) real 9.760 secs (user 1.530+27.270 sys 0.250+2.970)

tests: show cache of audited paths is never invalidated

Added signature for changeset 5544af862286

Added tag 4.3 for changeset 5544af862286

chg: define _GNU_SOURCE to allow CentOS 5 compilation Without this flag, compilation fails with: hgclient.c: In function 'hgc_open': hgclient.c:466: error: 'O_DIRECTORY' undeclared (first use in this function) hgclient.c:466: error: (Each undeclared identifier is reported only once hgclient.c:466: error: for each function it appears in.) Differential Revision: https://phab.mercurial-scm.org/D260

centos5: use vault urls CentOS 5 has reached end-of-life. A working build for it is still possible using 'vault.centos.org'. Differential Revision: https://phab.mercurial-scm.org/D261

ui: make sure buffer is flushed before waiting for user input (issue5587) Without this patch on Windows 'hg ci -i' hangs waiting for user input and "examine changes to 'file'? [Ynesfdaq?]" is never displayed (at least if the diff is sufficiently small). When Ctrl+C is pressed, this prompt becomes visible, which suggests that the buffer just wasn't flushed. I've never seen this happening on Linux, but this looks harmless enough to not platform-gate it.

util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) It appears that overriding __setitem__() doesn't work as documented on PyPy. Let's patch it as before bd0fd3ff9916. https://docs.python.org/2/library/collections.html#ordereddict-examples-and-recipes The issue was ui.configitems() wasn't ordered correctly, so the pull command was wrapped in different order.

test-revlog-raw: close file handles explicitly (issue5644)

tests: clear __pycache__ for PyPy compatibility (issue5638) (issue5642)

test-extension: glob module name in error message for PyPy

rebase: sort collapsed revisions in commit message (issue5643) This fixes the test instability.

i18n: use actual filename, in which function is defined, for hg.pot Before this patch, source filename for msgid in hg.pot file becomes incorrect, if a function is defined in file A, but detected in dict in file B, For example, almost all debug* commands are defined in debugcommands.py, but hggettext detects them in "table" of commands.py. Therefore, docstring fragments of debug* commands are marked as "defined in commands.py" in hg.pot file. This is serious problem for translation, because the cost to find out original location of texts increases very much.

ui: enable pager always for explicit --pager=on (issue5580) Before this patch, explicit --pager=on is unintentionally ignored by any disabling factor, even if priority of it is less than --pager=on (e.g. "[ui] paginate = off").

rebase: use one dirstateguard for when using rebase.singletransaction This was previously landed as 2519994d25ca but backed out in b63351f6a2 because it broke hooks mid-rebase and caused conflict resolution data loss in the event of unexpected exceptions. This new version adds the behavior back but behind a config flag, since the performance improvement is notable in large repositories. The old commit message was: Recently we switched rebases to run the entire rebase inside a single transaction, which dramatically improved the speed of rebases in repos with large working copies. Let's also move the dirstate into a single dirstateguard to get the same benefits. This let's us avoid serializing the dirstate after each commit. In a large repo, rebasing 27 commits is sped up by about 20%. I believe the test changes are because us touching the dirstate gave the transaction something to actually rollback. (grafted from 9e3dc3a1638b9754b58a0cb26aaa75d868058109) (grafted from 7d38b41d2266d9a02a15c64229fae0da5738dcec) Differential Revision: https://phab.mercurial-scm.org/D135

README: make paragraphs before example code end with "::" for reST syntax After this patch, README as reST text can be rendered into HTML more appropriately. This is useful for hosting servers, which render README file as a part of index HTML of its directory (for example, Bitbucket).

commit: don't let failed commit with --addremove update dirstate (issue5645) Differential Revision: https://phab.mercurial-scm.org/D204

tests: demonstrate that failed "hg ci -A" updates dirstate (issue5645) Differential Revision: https://phab.mercurial-scm.org/D203

tests: fix test-flagprocessor.t flakiness The test for duplicate flag processors depended on the timestamps being set in the dirstate to work. If the time between the the previous failed commit (which would set the timestamp, due to bug 5645) and the attempted commit with the duplicate flag processors was small enough, it would fail. The failure was caused by a call to commands.status() early in the commit process. If the dirstate did not have the timestamp set, it would need to fetch the file content to compare with. Since two flag processors had been registered, it would attempted to base64 decode the contents twice, which would of course fail. This patch adds a "hg debugrebuilddirstate" to make it deterministic and also replaces the test case's "hg commit" by simply "hg status", since that will trigger reading of the contents and thereby use of the flag processors as noted above. Differential Revision: https://phab.mercurial-scm.org/D202

tests: clarify that duplicate flag processors is not an error The flag processors test for duplicate processors for a single flag was misleading because the file from the previous test case caused it to fail (making the "echo 'this should fail' > file" part irrelevant). Let's remove the leftover from the previous test case to make it clear that duplicate flag processors results only in a warning. Note that duplicate flag processors would have resulted in a failure (not just a warning) until ea1c2eb7abd3 (extensions: catch uisetup and extsetup failures and don't let them break hg, 2017-06-06). I remember expressing my concern about ending up with half-loaded extensions. It would be pretty unfortunate to have double-encoded revlog content enter a repo, so maybe we should reconsider? Differential Revision: https://phab.mercurial-scm.org/D201

tests: avoid unexpected result at invocation of *.py file on Windows Invocation of "diff tool.py" in test-extdiff.t tests whether shellquote() is applied on specified command as expected. But direct invocation of "*.py" file might cause unexpected result on Windows according to suffix binding. For example, starting IDE, showing dialog to choose program to be used, and so on. In such case, running test-extdiff.t is easily timed out. This patch uses intermediate *.bat file on Windows, to avoid such unexpected result. Naming that intermediate file as "diff tool.bat" is enough to test applying shellquote().

tests: use pdiff instead of diff for portability "diff" command might cause redundant message, "No differences encountered" on Solaris for example. But suppressing option like "-q" isn't portable, because POSIX specification doesn't define it. pdiff script was introduced by b59ef0c21405 to stabilize output of standard diff command on each platforms.

tests: make pdiff return appropriate exit code at comparison of files Before this patch, pdiff script returns 0, even if diff is detected. This issue doesn't cause failure of tests using it, if it is invoked via extdiff extension, because extdiff itself examines changes between specified revisions and decides exit code. BTW, this patch ignores recursive comparison case, because: - there is no portable way for current while-read based implementation to return 1 at detecting changes - it isn't yet needed to replace direct "diff -r" invocation by pdiff for portability

merge with stable

tests: make test-profile.t pass if statprof didn't collect samples We had the first 3 tests in test-profile.t fail because the output didn't match. I have not yet confirmed that this was because no samples were collected, but we shouldn't require samples to be collected for the test to pass either way. Differential Revision: https://phab.mercurial-scm.org/D199

pypy: fix failing test-devel-warnings.t with Pypy5.6.0 In Pypy 5.6.0, traceback exception classes are not displayed with their full qualified name. Instead of displaying mercurial.error.ProgrammingError, Pypy displays ProgrammingError. Update the test to support both version.

pypy: fix failing test files with Pypy5.6.0 Pypy 5.6.0 saves cached bytecode files in __pycache__ directory, clean them in tests to fix loading old test extensions code. Doing so should also helps for Python3.x migration.

commitextras: make sure keys are not empty Thanks to Yuya for catching this.

make: add new directory 'hgdemandimport' to cleanbutpackages rule This fixes the build process for ppa (at least in my simple check). Perhaps we should loop through all directory that aren't 'packages'?

statichttprepo: implement wlock() (issue5613) statichttprepo inherits from localrepository. In doing so, it obtains default implementations of various methods, like wlock(). Before this change, tags cache writing would call repo.wlock(). This failed on statichttprepo due to localrepository's wlock() looking for an instance attribute that doesn't exist on statichttprepo (statichttprepo doesn't call localrepository.__init__). We /could/ define missing attributes until the base wlock() works. However, a statichttprepo is remote and read-only and can't be locked. The class already has a lock() that short circuits. So it makes sense to implement a short-circuited wlock() as well. That is what this patch does. LockError is expected to be raised when locking fails. The constructor takes a number of arguments that are local repository centric. Rather than rework LockError to not require them (which would not be appropriate for stable), this commit populates dummy values. I don't believe they'll ever be seen by the user, as lock failures on static http repos should be limited to well-defined (and tested) scenarios. We can and should revisit the LockError type to improve this.

test: add tests for branches and tags with static http repo (issue5613) Mercurial currently fails to clone by revision on static http repos with tags. The added tests demonstrate this.

commitextras: make sure keys contains ascii letters, numbers, '_' and '-' only

phabricator: convert unicode to binary when writing patches This is a quick fix to make `hg phabread D189` work. It seems we might want to replace all `r''` to `u''`, and add more `encoding.*to*` to be more explicit when interacting with `json` module. Differential Revision: https://phab.mercurial-scm.org/D192

setup: silence warning of unknown option python_requires on distutils /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'

setup: forcibly include cext/pure packages in py2exe (issue5625) Since these modules are implicitly imported by policy.importmod(), py2exe can't track them statically. cffi modules are excluded for now because they wouldn't be useful in frozen (i.e. CPython) environment.

setup: do not select hg executable that prints unexpected warnings Otherwise the subsequent hg.run() would fail. This factors out the filtering function so the same rule should apply.

docker: pass proxy arguments to docker process Building in Docker when behind a proxy requires passing the proxy variables. Differential Revision: https://phab.mercurial-scm.org/D190

makefile: build on CentOS 6 with a Python 2.7 Differential Revision: https://phab.mercurial-scm.org/D189

osx: wire up genosxversion script The only version strings that are changed are the ones baked into the .pkg - hg's self-reported version string doesn't change, so users will still see our mostly-pip-compatible version strings. For reference, the part of our versioning setup that's not PEP440 compatible is the RC releases - those should be .rc0 insted of -rc. It's too late to change that for the 4.3 cycle, so I'll worry about fixing that during the 4.4 cycle.

osx: new script for generating OS X package versions If you're shipping prerelease or rc packages using Munki, you'll eventually discover that Munki's version comparison logic is not as good as pip's. In theory we should be able to fix Munki, but it seems entirely reasonable to produce version strings that sort reasonably under these conditions. Since the requried logic not brief, add a new script and some tests of that logic. A followup change will wire this into the Makefile.

tests: update mac packaging test expectations Some more files are included these days.

tests: add support in run-tests.py for an environment variable to stop color I don't like the color output, so this gives me a way to easily never see it while still having pygments on my machine.

setup: fix mistake that prevented Python 3 from being excluded My intent was to only allow Python 3 if the right environment variable is set (for when people want to use `pip install .` on hg locally). This fixes the bug in my previous change. I verified that `python3.6 run-tests.py` still passes the tests that passed before, and that all tests pass on 2.7 (including our virtualenv-using installation test). Differential Revision: https://phab.mercurial-scm.org/D185

rebase: move bookmark to destination for commits becoming empty (issue5627) When rebasing a changeset X and that changeset becomes empty, we should move the bookmark on X to rebase destination. This is a regression caused by the scmutil.cleanupnodes refactoring for rebase. The `adjustdest` function calculates the destination of bookmark movement. It was back-ported from https://phab.mercurial-scm.org/D21. It might be slightly more powerful than the minimal requirement to solve this issue. For example, it's impossible for a merge changeset to become empty while any of its ancestors does not become empty, but the code could handle that case. Since the code is reasonably short and clean, and helps the upcoming D21 series, I'd like to check-in `adjustdest` now. Thanks Martin von Zweigbergk for spotting corner cases (-k and descendant with bookmarks) in this area!

setup: drop use of set literal so parsable on Python 2.6 Otherwise friendly error wouldn't be displayed.

setup: add extra logic to try and recommend a new pip on bad Python Modern pip can detect supported Python versions (which we now declare), and pull down a reasonable release. This trick was suggested in http://bit.ly/pycon2017-build-bridges, and seems like a good defensive maneuver so that when we want to move to Python 3 it's less risky for existing users. This moves the version-check logic after defining our printf function so we can print more informative messages.

setup: explicitly declare supported Python versions I think we should probably backport this to 4.2 as well, and do one more release there that explicitly declares 2.6 support. That way anyone stuck on Python 2.6 will end up getting the right hg if they use a modern pip to install. Users can still use `python setup.py` incantations to attempt installing Mercurial on unsupported Pythons, including 3.5 and 3.6. A followup change will switch to only doing our own Python-version-check logic if we're not being installed by a reasonable pip.

contrib: remove one test from the python 3 whitelist 439b4d005b4a introduced a use of 'hg revert --no-backup' which is currently broken. Rather than try and do a fix on the (complicated) revert code during the freeze, let's just give up on this test under Python 3 until later. Once we do fix things, the test ratchet script will re-enable it.

ui: fix configbytes isinstance check to look for bytes and not str Fixes configbytes on Python 3.

patch: update copying of dict keys and values to work on Python 3

phabricator: include the suggested arc config in the repo Test Plan: I uploaded this revision with `arc diff` Differential Revision: https://phab.mercurial-scm.org/D171

match: override visitdir() in nevermatcher to return False When we changed basematcher.visitdir() in cf15c3cc304c (match: make base matcher return True for visitdir, 2017-07-14), we forgot to add an override in nevermatcher. This led to tests failing in narrowhg. As Durham pointed out, it's high time to add unit tests for the matcher, so this patch also adds a first unit test. Differential Revision: https://phab.mercurial-scm.org/D151

wix: update list of necessary pyd files

run-tests: pre instantiate pygments objects This speeds up run-tests.py diff output by 10x, which affects developer experience significantly. As demonstrated by the following test: ``` #require pygments $ for i in `seq 1 200`; do > echo ' $ echo '$i >> test-a.t > echo ' wrong' >> test-a.t > done $ cat > walltime.py <<EOF > from __future__ import absolute_import, print_function > import os, sys, time > t1 = time.time() > os.system(' '.join(sys.argv[1:]) + ' >/dev/null 2>/dev/null') > t2 = time.time() > print('%0.2f' % (t2 - t1)) > EOF $ $PYTHON walltime.py $TESTDIR/run-tests.py -l --color=never $TESTTMP/test-a.t 0.24 Before this patch: $ $PYTHON walltime.py $TESTDIR/run-tests.py -l --color=always $TESTTMP/test-a.t 2.46 After this patch: $ $PYTHON walltime.py $TESTDIR/run-tests.py -l --color=always $TESTTMP/test-a.t 0.25 ```

contrib: drop Python 2.6 from manylinux1 wheel script

contrib: make build-linux-wheels.sh executable

test-dirstate-race: hide irrelevant hg status output See the explanation for more. Differential Revision: https://phab.mercurial-scm.org/D153

test-commit: stabilize for filesystems without symlink support

test-https: properly conditionalize Windows vs non-Windows output There's a bug in the (feature !) annotation, and this needs to be resolved before fixing it. Once fixed, the glob '*/missing' will match the output '/missing' on non-Windows platforms, consuming the output line and leaving nothing for the no-windows required line. I probably should have made (foo !) match only the 'foo' feature, and be ignored otherwise instead of optional. (re) didn't work here, because the parenthesis need to be escaped too.

Added signature for changeset 857876ebaed4

Added tag 4.3-rc for changeset 857876ebaed4

merge default into stable for code freeze

releasenotes: add custom admonitions support for release notes By default, the extension has default sections like fix, feature, perf etc.. This patch allow user to add support for custom admonition. In order to add a custom admonition, one needs to have a .hgreleasenotes file inside the repository. All the custom directive with name specified under the tag [sections] will be usable by the extension. One important thing to keep in mind is if there exists any custom admonitions with same key as default then they will override the default ones.

check-config: mention the file and line of the error I used this to more quickly track down a failing test-check-config.t issue in another repo. I thought it might be useful more generally, so I'm sending it out in case others think it's a worthwhile change.

rebase: add config to move rebase into a single transaction This was previously landed as cf8ad0e6c0e4 but backed out in a5abaa81fa because it broke hook mid rebase and caused conflict resolution data loss in the event of unexpected exceptions. This new version adds the behavior back but behind a config flag, since the performance improvement is notable in large repositories. The next patch adds a test covering this config. The old commit message was: Previously, rebasing would open several transaction over the course of rebasing several commits. Opening a transaction can have notable overhead (like copying the dirstate) which can add up when rebasing many commits. This patch adds a single large transaction around the actual commit rebase operation, with a catch for intervention which serializes the current state if we need to drop back to the terminal for user intervention. Amazingly, almost all the tests seem to pass. On large repos with large working copies, this can speed up rebasing 7 commits by 25%. I'd expect the percentage to be a bit larger for rebasing even more commits. There are minor test changes because we're rolling back the entire transaction during unexpected exceptions instead of just stopping mid-rebase, so there's no more backup bundle. It also leave an unknown file in the working copy, since our clean up 'hg update' doesn't delete unknown files. (grafted from cca36c7f35261b0e31beb226bf361067ef0e06ab) (grafted from dc497d8705b71503e32e07bd33925c1e42cf9c9a) Differential Revision: https://phab.mercurial-scm.org/D134

run-tests: remove unnecessary 'with_color' variable Its value is always the same as that of 'pygmentspresent'. Differential Revision: https://phab.mercurial-scm.org/D118

run-tests: warn if --color=always and no pygments installed Differential Revision: https://phab.mercurial-scm.org/D117

run-tests: drop unnecessary check for pygmentspresent 2893face0af5 (run-tests: check if stream is a tty before using color, 2017-07-18) made the check redundant but forgot to remove it. Differential Revision: https://phab.mercurial-scm.org/D116

run-tests: pass color option via test case object , not global var At first I updated the color field of the 'options' object (from the CLI parser), but then I decided to put it directly on the test case object itself to avoid mutating the shared object (even though all tests would have the same value). Differential Revision: https://phab.mercurial-scm.org/D114

phabricator: sanity check Differential Revision from commit message Previously, we trust Differential Revision in commit message blindly. This patch adds sanity check so a host name change will be detected and the commit message will be ignored. Differential Revision: https://phab.mercurial-scm.org/D35

test-phases-exchange: properly conditionalize output I forgot to invert the conditions, and the test runner didn't catch it[1]. But since only '&&' is supported in the per-line feature tests, this won't work anyway. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-July/101941.html

commitextras: mark the extension as ADVANCED

run-tests: check if stream is a tty before using color Previous implementation (e80041832eec) checked only if sys.stderr was a tty which was less general. Also makes sure that colors is never used if pygments is not available, irrespective of --color flag value.

tests: rewrite test-rebase-detach.t using drawdag This makes it much clearer because each test case can have exactly the graph it wants (no extra cruft just because other tests want it). Differential Revision: https://phab.mercurial-scm.org/D94

tests: simplify config in test-rebase-detach.t Differential Revision: https://phab.mercurial-scm.org/D93

drawdag: include files from both parents in merge commits Consider a graph like this: D |\ B C |/ A drawdag will add a file called A in commit A, file B in B, file C in C. That's fine and expected. In merge commits like D, I would expect the files and their contents to be taken from the parent commits, so commit D in this example would have files A, B, and C. However, drawdag will instead add the file D compared to the first parent. Depending on whether B or C got a smaller nodeid, the contents of D would be {A, B, D} or {A, C, D}. This patch changes it to to be {A, B, C}. Differential Revision: https://phab.mercurial-scm.org/D92

gitweb: preserve whitespace in description Without this, multiple spaces or tabs in the commit message aren't preserved and things like tables don't align properly. As part of adding the CSS rule, we had to cuddle the content with the <div> to not introduce leading and trailing whitespace. The "addbreaks" filter was also removed because it would insert an additional newline, effectively double spacing content. Differential Revision: https://phab.mercurial-scm.org/D113

sparse: add a requirement when a repository uses sparse (BC) The presence of a sparse checkout can confuse legacy clients or clients without sparse enabled for reasons that should be obvious. This commit introduces a new repository requirement that tracks whether sparse is enabled. The requirement is added when a sparse config is activated and removed when the sparse config is reset. The localrepository constructor has been taught to not open repos with this requirement unless the sparse feature is enabled. It yields a more actionable error message than what you would get if the lockout were handled strictly at the requirements verification phase. Old clients that aren't sparse aware will see the generic "repository requires features unknown to this Mercurial" error, however. The new requirement has "exp" in its name to reflect the experimental nature of sparse. There's a chance that the eventual non-experimental feature won't change significantly and we could have squatted on the "sparse" requirement without ill effect. If that happens, we can teach new clients to still recognize the old name. But I suspect we'll sneak in some BC and we'll want a new requirement to convey new meaning. Differential Revision: https://phab.mercurial-scm.org/D110

sparse: consolidate common code for writing sparse config In 3 functions we were writing the sparse config and updating the working directory. In two of them we had a transaction-like process for restoring the sparse config in case of wdir update fail. Because the pattern is common, we've already made mistakes, and the complexity will increase in the near future, let's consolidate the code into a reusable function. As part of this refactor, we end up reading the "sparse" file twice when updating it. This is a bit sub-optimal. But I don't think it is worth the code complexity to pass around the variables to avoid the redundancy. Differential Revision: https://phab.mercurial-scm.org/D109

revset: pass repo when passing ui The repo instance is currently only used to provide a changeset lookup function as part of parsing revsets. I /think/ this allows node fragments to resolve. I'm not sure why we wouldn't want this to always "just work" if parsing a revset string. Plus, an upcoming commit will introduce a new consumer that needs a handle on the repo. So passing it more often will make that code work more. Passing a repo instance in all callers of revset.match* results in a bunch of test changes. Notably, branch and tags caches get populated as part of evaluating revsets. I'm not sure if this is desirable. So this patch takes the conservative approach and only passes the repo if we're passing a ui instance. Differential Revision: https://phab.mercurial-scm.org/D97

win32: copy-edit debugssl messages to match prevailing style

run-tests: make sure to check if pygments is installed before using it e80041832e introduced support to color the output of tests but used pygments without checking whether it's installed or not. That breaks test-run-tests.t for machines which don't have pygments installed. This patch conditionalize the color test in test-run-tests.t and also add a check to make sure pygments is installed before using that.

sparse: require [section] in sparse config files (BC) Previously, [include] was implicit and pattern lines before a [section] were added to includes. Because the format may change in the future and explicit behavior, well, more explicit, this commit changes the config parser to reject pattern lines that don't occur in a [section]. Differential Revision: https://phab.mercurial-scm.org/D96

sparse: use set for capturing profiles Order doesn't need to be preserved. A set is acceptable. Differential Revision: https://phab.mercurial-scm.org/D95

util: remove dead code which used to be for old python2 versions Differential Revision: https://phab.mercurial-scm.org/D107

status: add a flag to terse the output (issue4119) This adds an experimental flag -t/--terse which will terse the output. The terse flag will respect other flags which filters the output. The flag takes a string whose value can be a subsequence of "marduic" (the order does not matter here.) Ignored files are not considered while tersing unless -i flag is passed or 'i' is there is the terse flag value. The flag is experimental for testing as there may be cases which will produce strange results with the flag. We can set the terse on by default by simply passing 'u' to the cmdutil.tersestatus(). This patch also adds a test file with tests covering the new feature.

commitextras: check the format of the arguments and no internal key is used This patch adds check to make the arguments are passed as KEY=VALUE and no key which is used internally is passed. This patch also adds test for the extension.

commitextras: move fb extension to core which add extras to a commit This patch moves the Facebook extension to add extra fields to a commit to a in-core extension.

archive: add an experimental config to control the metadata file template Experimental because given the possible complexity, it may be worth figuring out how to load this from a file, similar to the style files for the log command, instead of trying to stuff it on the command line.

archive: use a templater to build the metadata file There are no visible changes here. I'm starting to wonder if adding the '+' to the 'node' line instead of a separate key line in 3047167733dc was the right thing to do. The '{node}' keyword never includes '+' elsewhere, and the way setup.py works, it would truncate it anyway. Additionally, the file is missing '{p2node}' when 'wdir()' merges are archived. I thought about adding an 'identify' line that would correspond to `hg id -n`. But the other nodes are the full 40 characters, and the output most useful for versioning is the short form. All of this cries out for customization via templating. (Although maybe having the short identify line by default is still a good idea.)

commandserver: do not handle EINTR for selector.select selectors2 library handles EINTR transparently so we don't need to handle select.error ourselves.

debugobsolete: also report the number of obsoleted changesets This seems useful to have the number of obsoleted changesets when calling debugobsolete.

transaction-summary: display the summary for all transactions Now that we records "all" changes happening in a transaction (in tr.changes) we will be able to provide better report on various changes (phases turned public, changeset obsoleted, branch merged or created, etc..) This is far too late in the cycle to play with this, but having this existing method called more widely will help extensions to play around with various options during the 4.4 cycle. Instead of calling registersummarycallback only for transactions we want, we always call it and use the transaction name to decide when to report (eg: we do not want `hg amend` to report new obsoleted changesets). Filtering on transaction name does not seems great, but seems good enough for the moment. We can change the API during the next cycle. The previous manual call during unbundling of the bundle2 "obsmarkers" part is no longer necessary and has been dropped.

test-strip: fix call to super class in crashstriprepo The 'self' argument is already taken care of by the 'super(...)' call. Instead we were passing the repository instance as the 'desc' value. This got discovered by breaking the next changeset.

share: share 'cachevfs' with the source clone (issue5108) Share extension now also share caches reads and writes. Not sharing caches results in costly caches recomputations which can takes up to minutes when using shares on large repositories. There are a couple of file in the '.hg/cache/' that depends of the current visibility. Visibility can be affected by the working copy location, something which is specific to each share. We ignores them for this series because they: * are the minority, * already have a good fallback to other precomputed caches, * are only affected when people use the experimental evolution feature.

cachevfs: add a devel warning for cache access though 'vfs' This will help third party extensions to migrate to the new 'cachevfs'.

cachevfs: migration the tags fnode cache to 'cachevfs' This will help sharing the cache between shares.

cachevfs: migrate tagscache to 'cachevfs' This will help sharing the cache between shares.

cachevfs: migration the revbranchcache to 'cachevfs' This will help sharing the cache between shares.

cachevfs: use the new vfs in when computing branchmap cache This will help sharing the cache between shares.

cachevfs: add a vfs dedicated to cache Most of the cache content lives in '.hg/cache/'. Moreover they are computed exclusively from data in the '.hg/store' directory. This creates issues with the share extension as the '.hg/store' directory is shared but the '.hg/cache' is not. On large repositories, this makes this prevent some usage of the share extension inefficient as some caches can take minutes to be recomputed. To improve the situation, we introduce a new 'cachevfs' that will be dedicated to cache reading and writing. In the next patches of this series, we'll migrate the 4 existing caches to it and update the share extension.

vfsward: register 'write with no lock' warnings as 'check-locks' config Update 'write with no lock' warnings in order to be better controlled by the config. We reuse the option used for lock order for these other lock related message. The message can now be disabled using 'devel.check-locks = no' (in addition to the usual 'devel.all-warnings = no').

demandimport: prefer loaded module over package attribute (issue5617) In general, the attribute of the same name is overwritten by executing an import statement. import a.b print(a.b.c) # 'c' of a/b/__init__.py from a.b.c import d print(a.b.c) # a/b/c.py However, this appears not true for the scenario described in the test case, and surprisingly, "from a.b.c import d" works even if "a.b.c" is not a module. This patch works around the problem by taking the right module from sys.modules if available.

demandimport: drop Py3 workarounds from Py2 implementation

demandimport: drop hack for old Pythons which had no level argument _import() was added by 868282fa29d8, but it is identical to _origimport() now.

tests: drop "absimport" conditionals which should be always true

color: drop the now useless color extension all the extension features are provided by core since 4.2.

extensions: expand the builtins extensions declaration This will future updates of the set cleaner and more readable.

debugextensions: drop the color from the test We are about to remove the extension so we remove trace of it in this specific test to prevent it to be confused.

configitems: register the 'bugzilla.notify' config

configitems: register the 'ui.mergemarkertemplate' config

configitems: register the 'ui.color' config

color: drop the _enabledbydefault module variable Since color is on by default, cleanup the now useless variable in both core and color extension.

configitems: register the 'ui.forcecwd' config

configitems: register the 'ui.fallbackencoding' config

bookmark: deprecate direct del of a bookmark value We want all bookmark deletion to go through 'applychanges', so lets deprecate legacy ways of doing bookmark deletion.

bookmark: deprecate direct set of a bookmark value We want all bookmark update to go through 'applychanges', so lets deprecate legacy ways of doing bookmark update.

bookmark: track bookmark changes at the transaction level The transaction has now a 'bookmarks' dictionary in tr.changes. The structure of the dictionary is {BOOKMARK_NAME: (OLD_NODE, NEW_NODE)}. If a bookmark is deleted NEW_NODE will be None. If a bookmark is created OLD_NODE will be None. If the bookmark is updated multiple time, the initial value is preserved.

bookmark: deprecate 'recordchange' in favor of 'applychanges' Now that we have migrated all in-core caller of 'recordchange' to 'applychanges', deprecate 'recordchange' so external callers will move to the new unified method.

bookmark: drop deletedivergent It has no caller anymore.

bookmark: use 'divergent2delete' in checkconflict checkconflict used to also do some bookmark deletion in case of divergence. It is a bit suspicious given the function name, but it's not the goal of this series. In order to unify bookmarks changing, checkconflict now return the list of divergent bookmarks to clean up and the callers must clean them by calling applyphases.

bookmark: use 'divergent2delete' when updating a bookmark

bookmark: use 'divergent2delete' in 'scmutil.cleanupnode'

bookmark: split out target computation from 'deletedivergent' We want to use applychanges in order to unify bookmark movement. We need a way to compute divergence deletion without actually removing them. We split the function in two in this patch while we migrate the existing users of this code on next patches.

bookmark: remove a useless 'recordchange' in the amend code We do not touch the bookmarks store in this code, just the active bookmark, not covered by the transaction. So it seems we can safely drop this call and the tests agree with us.

test-phase-exchange: stabilize for Windows

debugignore: eliminate inconsistencies with `hg status` (issue5222) Using a matcher for this command allows processing the named file(s) as relative to cwd. It also leverages the icasefs normalization logic the same way the status command does. (However, a false indicator is given for a nonexistent file in some cases, e.g. passing 'foo.REJ' when that file doesn't exist, and the rule is '*.rej'. Maybe the regex itself needs to be case insensitive on these platforms, at least for the debug command.) Finally, the file printed is relative to cwd and uses platform specific slashes, so a few (glob)s were needed in seemingly unrelated tests.

commandserver: close selector explicitly The selector does not have a __del__ method and needs a manual close. We can also use "with selector" but that makes the code too indented. Therefore append a "selector.close()" after the end of the main loop for now.

scmutil: remove duplicated import of i18n._()

obsstore: let read marker API take a range of offsets This allows us to read a customized range of markers, instead of loading all of them. The condition of stop is made consistent across C and Python implementation so we will still read marker when offset=a, stop=a+1.

commandserver: use selectors2 Previously, commandserver was using select.select. That could have issue if _sock.fileno() >= FD_SETSIZE (usually 1024), which raises: ValueError: filedescriptor out of range in select() We got that in production today, although it's the code opening that many files to blame, it seems better for commandserver to work in this case. There are multiple way to "solve" it, like preserving a fd with a small number and swap it with sock using dup2(). But upgrading to a modern selector supported by the system seems to be the most correct way.

selector2: vendor selector2 library This library was a backport of the Python 3 "selectors" library. It is useful to provide a better selector interface for Python2, to address some issues of the plain old select.select, mentioned in the next patch. The code [1] was ported using the MIT license, with some minor modifications to make our test happy: 1. "# no-check-code" was added since it's foreign code. 2. "from __future__ import absolute_import" was added. 3. "from collections import namedtuple, Mapping" changed to avoid direct symbol import. [1]: https://github.com/SethMichaelLarson/selectors2/blob/d27dbd2fdc48331fb76ed431f44b6e6956de7f82/selectors2.py # no-check-commit

context: name files relative to cwd in warning messages I was several directories deep in the kernel tree, ran `hg add`, and got the warning about the size of one of the files. I noticed that it suggested undoing the add with a specific revert command. The problem is, it would have failed since the path printed was relative to the repo root instead of cwd. While here, I just fixed the other messages too. As an added benefit, these messages now look the same as the verbose/inexact messages for the corresponding command. I don't think most of these messages are reachable (typically the corresponding cmdutil function does the check). I wasn't able to figure out why the keyword tests were failing when using pathto()- I couldn't cause an absolute path to be used by manipulating the --cwd flag on a regular add. (I did notice that keyword is adding the file without holding wlock.)

run-tests: disable color on Windows More Windows sadness. Maybe someone can figure out how to make win32 color work, but I think we avoid importing stuff from the mercurial package in this module. On the plus side, this conditionalizes away a test failure.

codemod: register core configitems using a script This is done by a script [2] using RedBaron [1], a tool designed for doing code refactoring. All "default" values are decided by the script and are strongly consistent with the existing code. There are 2 changes done manually to fix tests: [warn] mercurial/exchange.py: experimental.bundle2-output-capture: default needs manual removal [warn] mercurial/localrepo.py: experimental.hook-track-tags: default needs manual removal Since RedBaron is not confident about how to indent things [2]. [1]: https://github.com/PyCQA/redbaron [2]: https://github.com/PyCQA/redbaron/issues/100 [3]: #!/usr/bin/env python # codemod_configitems.py - codemod tool to fill configitems # # Copyright 2017 Facebook, Inc. # # This software may be used and distributed according to the terms of the # GNU General Public License version 2 or any later version. from __future__ import absolute_import, print_function import os import sys import redbaron def readpath(path): with open(path) as f: return f.read() def writepath(path, content): with open(path, 'w') as f: f.write(content) _configmethods = {'config', 'configbool', 'configint', 'configbytes', 'configlist', 'configdate'} def extractstring(rnode): """get the string from a RedBaron string or call_argument node""" while rnode.type != 'string': rnode = rnode.value return rnode.value[1:-1] # unquote, "'str'" -> "str" def uiconfigitems(red): """match *.ui.config* pattern, yield (node, method, args, section, name)""" for node in red.find_all('atomtrailers'): entry = None try: obj = node[-3].value method = node[-2].value args = node[-1] section = args[0].value name = args[1].value if (obj in ('ui', 'self') and method in _configmethods and section.type == 'string' and name.type == 'string'): entry = (node, method, args, extractstring(section), extractstring(name)) except Exception: pass else: if entry: yield entry def coreconfigitems(red): """match coreconfigitem(...) pattern, yield (node, args, section, name)""" for node in red.find_all('atomtrailers'): entry = None try: args = node[1] section = args[0].value name = args[1].value if (node[0].value == 'coreconfigitem' and section.type == 'string' and name.type == 'string'): entry = (node, args, extractstring(section), extractstring(name)) except Exception: pass else: if entry: yield entry def registercoreconfig(cfgred, section, name, defaultrepr): """insert coreconfigitem to cfgred AST section and name are plain string, defaultrepr is a string """ # find a place to insert the "coreconfigitem" item entries = list(coreconfigitems(cfgred)) for node, args, nodesection, nodename in reversed(entries): if (nodesection, nodename) < (section, name): # insert after this entry node.insert_after( 'coreconfigitem(%r, %r,\n' ' default=%s,\n' ')' % (section, name, defaultrepr)) return def main(argv): if not argv: print('Usage: codemod_configitems.py FILES\n' 'For example, FILES could be "{hgext,mercurial}/*/**.py"') dirname = os.path.dirname reporoot = dirname(dirname(dirname(os.path.abspath(__file__)))) # register configitems to this destination cfgpath = os.path.join(reporoot, 'mercurial', 'configitems.py') cfgred = redbaron.RedBaron(readpath(cfgpath)) # state about what to do registered = set((s, n) for n, a, s, n in coreconfigitems(cfgred)) toregister = {} # {(section, name): defaultrepr} coreconfigs = set() # {(section, name)}, whether it's used in core # first loop: scan all files before taking any action for i, path in enumerate(argv): print('(%d/%d) scanning %s' % (i + 1, len(argv), path)) iscore = ('mercurial' in path) and ('hgext' not in path) red = redbaron.RedBaron(readpath(path)) # find all repo.ui.config* and ui.config* calls, and collect their # section, name and default value information. for node, method, args, section, name in uiconfigitems(red): if section == 'web': # [web] section has some weirdness, ignore them for now continue defaultrepr = None key = (section, name) if len(args) == 2: if key in registered: continue if method == 'configlist': defaultrepr = 'list' elif method == 'configbool': defaultrepr = 'False' else: defaultrepr = 'None' elif len(args) >= 3 and (args[2].target is None or args[2].target.value == 'default'): # try to understand the "default" value dnode = args[2].value if dnode.type == 'name': if dnode.value in {'None', 'True', 'False'}: defaultrepr = dnode.value elif dnode.type == 'string': defaultrepr = repr(dnode.value[1:-1]) elif dnode.type in ('int', 'float'): defaultrepr = dnode.value # inconsistent default if key in toregister and toregister[key] != defaultrepr: defaultrepr = None # interesting to rewrite if key not in registered: if defaultrepr is None: print('[note] %s: %s.%s: unsupported default' % (path, section, name)) registered.add(key) # skip checking it again else: toregister[key] = defaultrepr if iscore: coreconfigs.add(key) # second loop: rewrite files given "toregister" result for path in argv: # reconstruct redbaron - trade CPU for memory red = redbaron.RedBaron(readpath(path)) changed = False for node, method, args, section, name in uiconfigitems(red): key = (section, name) defaultrepr = toregister.get(key) if defaultrepr is None or key not in coreconfigs: continue if len(args) >= 3 and (args[2].target is None or args[2].target.value == 'default'): try: del args[2] changed = True except Exception: # redbaron fails to do the rewrite due to indentation # see https://github.com/PyCQA/redbaron/issues/100 print('[warn] %s: %s.%s: default needs manual removal' % (path, section, name)) if key not in registered: print('registering %s.%s' % (section, name)) registercoreconfig(cfgred, section, name, defaultrepr) registered.add(key) if changed: print('updating %s' % path) writepath(path, red.dumps()) if toregister: print('updating configitems.py') writepath(cfgpath, cfgred.dumps()) if __name__ == "__main__": sys.exit(main(sys.argv[1:]))

phabricator: allow specifying reviewers on phabsend Sometimes people want to specify reviewer explicitly for a stack. The webpage only allows changing reviewer for one revision at a time. This patch adds a `--reviewer` flag to make it easier to specify reviewers. Test Plan: On a test Phabricator instance, enable `differential.allow-self-accept`, assign myself as a reviewer and make sure it works. Also try an invalid username and make sure it raises. Differential Revision: https://phab.mercurial-scm.org/D38

match: remove unused negatematcher This was only used by the sparse extension's dirstate._ignore override, which no longer exists. Differential Revision: https://phab.mercurial-scm.org/D60

sparse: override dirstate.walk() instead of dirstate._ignore Instead of treating files that are outside the sparse config as ignored, this makes it so we list only those that are within the sparse config by passing the sparse matcher to dirstate.walk(). Once we add support for narrow (sparseness applied to history, not just working copy), we will need to do a similar restriction of the walk over manifests, so this will be more consistent then. It also simplifies the code a bit. Note that a side-effect of this change is that files outside the sparse config used to be listed as ignored, but they will now not be listed at all. This can be seen in the test case where "hg purge" no longer has any effect because it doesn't see that the files outside the space config exist. To fix that, I think we should add an option to dirstate.walk() to walk outside the sparse config. We might expose that to the user as --no-sparse flag to e.g. "hg status" and "hg purge", but that's work for another day. Differential Revision: https://phab.mercurial-scm.org/D59

patch: use devel.all-warnings to replace devel.all It appears to be a misspell in patch.py.

sslutil: inform the user about how to fix an incomplete certificate chain This is a Windows only thing. Unfortunately, the socket is closed at this point (so the certificate is unavailable to check the chain). That means it's printed out when verification fails as a guess, on the assumption that 1) most of the time verification won't fail, and 2) sites using expired or certs that are too new will be rare. Maybe this is an argument for adding more functionality to debugssl, to test for problems and print certificate info. Or maybe it's an argument for bundling certificates with the Windows builds. That idea was set aside when the enhanced SSL code went in last summer, and it looks like there were issues with using certifi on Windows anyway[1]. This was tested by deleting the certificate out of certmgr.msc > "Third-Party Root Certification Authorities" > "Certificates", seeing `hg pull` fail (with the new message), trying this command, and then successfully performing the pull command. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-October/089573.html

debug: add a method to check the state of, and built an SSL cert chain This is only useful on Windows, and avoids the need to use Internet Explorer to build the certificate chain. I can see this being extended in the future to print information about the certificate(s) to help debug issues on any platform. Maybe even perform some of the python checks listed on the secure connections wiki page. But for now, all I need is 1) a command that can be invoked in a setup script to ensure the certificate is installed, and 2) a command that the user can run if/when a certificate changes in the future. It would have been nice to leverage the sslutil library to pick up host specific settings, but attempting to use sslutil.wrapsocket() failed the 'not sslsocket.cipher()' check in it and aborted. The output is a little more chatty than some commands, but I've seen the update take 10+ seconds, and this is only a debug command.

win32: add a method to trigger the Crypto API to complete a certificate chain I started a thread[1] on the mailing list awhile ago, but the short version is that Windows doesn't ship with a full list of certificates[2]. Even if the server sends the whole chain, if Windows doesn't have the appropriate certificate pre-installed in its "Third-Party Root Certification Authorities" store, connections mysteriously fail with: abort: error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661) Windows expects the application to call the methods invoked here as part of the certificate verification, triggering a call out to Windows update if necessary, to complete the trust chain. The python bug to add this support[3] hasn't had any recent activity, and isn't targeting py27 anyway. The only work around that I could find (besides figuring out the certificate and walking through the import wizard) is to browse to the site in Internet Explorer. Opening the page with FireFox or Chrome didn't work. That's a pretty obscure way to fix a pretty obscure problem. We go to great lengths to demystify various SSL errors, but this case is clearly lacking. Let's try to make things easier to diagnose and fix. When I had trouble figuring out how to get ctypes to work with all of the API pointers, I found that there are other python projects[4] using this API to achieve the same thing. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-April/096501.html [2] https://support.microsoft.com/en-us/help/931125/how-to-get-a-root-certificate-update-for-windows [3] https://bugs.python.org/issue20916 [4] https://github.com/nvaccess/nvda/blob/3b86bce2066b1934df14b96f2e83369900860ecf/source/updateCheck.py#L511

bookmarks: use 'applychanges' for bookmark update There is still some use of 'deletedivergent' bookmark here. They will be taken care of later. The 'deletedivergent' code needs some rework before fitting in the new world.

bookmark: use 'applychanges' in 'repair.strip'

bookmark: use 'applychanges' in the mq extension

bookmark: use 'applychanges' when stripping

bookmark: use 'applychanges' in the convert extension