pull: add tests for unsafe ssh url (SEC)

clone: add tests for unsafe ssh url (SEC)

sshpeer: check for safe ssh url (SEC) Checking in the sshpeer for a rogue ssh:// urls seems like the right place to do it (instead of whack-a-mole with pull, clone, push, etc).

ssh: ban any username@host or host that starts with - (SEC) This paranoia probably isn't required, but it can't hurt either.

util: add utility method to check for bad ssh urls (SEC) Our use of SSH has an exploit that will parse the first part of an url blindly as a hostname. Prior to this set of security patches, a url with '-oProxyCommand' could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' can be abused to execute arbitrary commands in a similar fashion. We defend against this by checking ssh:// URLs and looking for a hostname that starts with a - or contains a |. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

pathauditor: disable cache of audited paths by default (issue5628) The initial attempt was to discard cache when appropriate, but it appears to be error prone. We had to carefully inspect all places where audit() is called e.g. without actually updating filesystem, before removing files and directories, etc. So, this patch disables the cache of audited paths by default, and enables it only for the following cases: - short-lived auditor objects - repo.vfs, repo.svfs, and repo.cachevfs, which are managed directories and considered sort of append-only (a file/directory would never be replaced with a symlink) There would be more cacheable vfs objects (e.g. mq.queue.opener), but I decided not to inspect all of them in this patch. We can make them cached later. Benchmark result: - using old clone of http://selenic.com/repo/linux-2.6/ (38319 files) - on tmpfs - run HGRCPATH=/dev/null hg up -q --time tip && hg up -q null - try 4 times and take the last three results original: real 7.480 secs (user 1.140+22.760 sys 0.150+1.690) real 8.010 secs (user 1.070+22.280 sys 0.170+2.120) real 7.470 secs (user 1.120+22.390 sys 0.120+1.910) clearcache (the other series): real 7.680 secs (user 1.120+23.420 sys 0.140+1.970) real 7.670 secs (user 1.110+23.620 sys 0.130+1.810) real 7.740 secs (user 1.090+23.510 sys 0.160+1.940) enable cache only for vfs and svfs (this series): real 8.730 secs (user 1.500+25.190 sys 0.260+2.260) real 8.750 secs (user 1.490+25.170 sys 0.250+2.340) real 9.010 secs (user 1.680+25.340 sys 0.280+2.540) remove cache function at all (for reference): real 9.620 secs (user 1.440+27.120 sys 0.250+2.980) real 9.420 secs (user 1.400+26.940 sys 0.320+3.130) real 9.760 secs (user 1.530+27.270 sys 0.250+2.970)

tests: show cache of audited paths is never invalidated

Added signature for changeset 5544af862286

Added tag 4.3 for changeset 5544af862286

chg: define _GNU_SOURCE to allow CentOS 5 compilation Without this flag, compilation fails with: hgclient.c: In function 'hgc_open': hgclient.c:466: error: 'O_DIRECTORY' undeclared (first use in this function) hgclient.c:466: error: (Each undeclared identifier is reported only once hgclient.c:466: error: for each function it appears in.) Differential Revision: https://phab.mercurial-scm.org/D260

centos5: use vault urls CentOS 5 has reached end-of-life. A working build for it is still possible using 'vault.centos.org'. Differential Revision: https://phab.mercurial-scm.org/D261

ui: make sure buffer is flushed before waiting for user input (issue5587) Without this patch on Windows 'hg ci -i' hangs waiting for user input and "examine changes to 'file'? [Ynesfdaq?]" is never displayed (at least if the diff is sufficiently small). When Ctrl+C is pressed, this prompt becomes visible, which suggests that the buffer just wasn't flushed. I've never seen this happening on Linux, but this looks harmless enough to not platform-gate it.

util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) It appears that overriding __setitem__() doesn't work as documented on PyPy. Let's patch it as before bd0fd3ff9916. https://docs.python.org/2/library/collections.html#ordereddict-examples-and-recipes The issue was ui.configitems() wasn't ordered correctly, so the pull command was wrapped in different order.

test-revlog-raw: close file handles explicitly (issue5644)

tests: clear __pycache__ for PyPy compatibility (issue5638) (issue5642)

test-extension: glob module name in error message for PyPy

rebase: sort collapsed revisions in commit message (issue5643) This fixes the test instability.

i18n: use actual filename, in which function is defined, for hg.pot Before this patch, source filename for msgid in hg.pot file becomes incorrect, if a function is defined in file A, but detected in dict in file B, For example, almost all debug* commands are defined in debugcommands.py, but hggettext detects them in "table" of commands.py. Therefore, docstring fragments of debug* commands are marked as "defined in commands.py" in hg.pot file. This is serious problem for translation, because the cost to find out original location of texts increases very much.

ui: enable pager always for explicit --pager=on (issue5580) Before this patch, explicit --pager=on is unintentionally ignored by any disabling factor, even if priority of it is less than --pager=on (e.g. "[ui] paginate = off").

rebase: use one dirstateguard for when using rebase.singletransaction This was previously landed as 2519994d25ca but backed out in b63351f6a2 because it broke hooks mid-rebase and caused conflict resolution data loss in the event of unexpected exceptions. This new version adds the behavior back but behind a config flag, since the performance improvement is notable in large repositories. The old commit message was: Recently we switched rebases to run the entire rebase inside a single transaction, which dramatically improved the speed of rebases in repos with large working copies. Let's also move the dirstate into a single dirstateguard to get the same benefits. This let's us avoid serializing the dirstate after each commit. In a large repo, rebasing 27 commits is sped up by about 20%. I believe the test changes are because us touching the dirstate gave the transaction something to actually rollback. (grafted from 9e3dc3a1638b9754b58a0cb26aaa75d868058109) (grafted from 7d38b41d2266d9a02a15c64229fae0da5738dcec) Differential Revision: https://phab.mercurial-scm.org/D135

README: make paragraphs before example code end with "::" for reST syntax After this patch, README as reST text can be rendered into HTML more appropriately. This is useful for hosting servers, which render README file as a part of index HTML of its directory (for example, Bitbucket).

commit: don't let failed commit with --addremove update dirstate (issue5645) Differential Revision: https://phab.mercurial-scm.org/D204

tests: demonstrate that failed "hg ci -A" updates dirstate (issue5645) Differential Revision: https://phab.mercurial-scm.org/D203

tests: fix test-flagprocessor.t flakiness The test for duplicate flag processors depended on the timestamps being set in the dirstate to work. If the time between the the previous failed commit (which would set the timestamp, due to bug 5645) and the attempted commit with the duplicate flag processors was small enough, it would fail. The failure was caused by a call to commands.status() early in the commit process. If the dirstate did not have the timestamp set, it would need to fetch the file content to compare with. Since two flag processors had been registered, it would attempted to base64 decode the contents twice, which would of course fail. This patch adds a "hg debugrebuilddirstate" to make it deterministic and also replaces the test case's "hg commit" by simply "hg status", since that will trigger reading of the contents and thereby use of the flag processors as noted above. Differential Revision: https://phab.mercurial-scm.org/D202

tests: clarify that duplicate flag processors is not an error The flag processors test for duplicate processors for a single flag was misleading because the file from the previous test case caused it to fail (making the "echo 'this should fail' > file" part irrelevant). Let's remove the leftover from the previous test case to make it clear that duplicate flag processors results only in a warning. Note that duplicate flag processors would have resulted in a failure (not just a warning) until ea1c2eb7abd3 (extensions: catch uisetup and extsetup failures and don't let them break hg, 2017-06-06). I remember expressing my concern about ending up with half-loaded extensions. It would be pretty unfortunate to have double-encoded revlog content enter a repo, so maybe we should reconsider? Differential Revision: https://phab.mercurial-scm.org/D201

tests: avoid unexpected result at invocation of *.py file on Windows Invocation of "diff tool.py" in test-extdiff.t tests whether shellquote() is applied on specified command as expected. But direct invocation of "*.py" file might cause unexpected result on Windows according to suffix binding. For example, starting IDE, showing dialog to choose program to be used, and so on. In such case, running test-extdiff.t is easily timed out. This patch uses intermediate *.bat file on Windows, to avoid such unexpected result. Naming that intermediate file as "diff tool.bat" is enough to test applying shellquote().

tests: use pdiff instead of diff for portability "diff" command might cause redundant message, "No differences encountered" on Solaris for example. But suppressing option like "-q" isn't portable, because POSIX specification doesn't define it. pdiff script was introduced by b59ef0c21405 to stabilize output of standard diff command on each platforms.

tests: make pdiff return appropriate exit code at comparison of files Before this patch, pdiff script returns 0, even if diff is detected. This issue doesn't cause failure of tests using it, if it is invoked via extdiff extension, because extdiff itself examines changes between specified revisions and decides exit code. BTW, this patch ignores recursive comparison case, because: - there is no portable way for current while-read based implementation to return 1 at detecting changes - it isn't yet needed to replace direct "diff -r" invocation by pdiff for portability

merge with stable

tests: make test-profile.t pass if statprof didn't collect samples We had the first 3 tests in test-profile.t fail because the output didn't match. I have not yet confirmed that this was because no samples were collected, but we shouldn't require samples to be collected for the test to pass either way. Differential Revision: https://phab.mercurial-scm.org/D199