ui: add option to timestamp status and diagnostic messages Differential Revision: https://phab.mercurial-scm.org/D8640

tests: use path inside test dir This will make the diff for the next patch less noisy.

convert: convert URLs to UTF-8 for Subversion Preamble: for comprehension, note that the `path` of geturl() would better be called `path_or_url` (the argument of the call of getsvn() is called `url`). For HTTP(S) URLs, the changes don’t make a difference, as they are restricted to ASCII. For file URLs, the reasoning is the same as for paths: we have to roundtrip with what Subversion is doing. When the locale encoding is ISO-8859-15, trying to convert a SVN repo `file:///tmp/a€` failed before like this: file:///tmp/a%A4 does not look like a Subversion repository to libsvn version 1.14.0 Decoding the path using the locale encoding can fail. In this case, we have to bail out, as Subversion won’t be able to do anything useful with the path.

convert: correctly convert paths to UTF-8 for Subversion The previous code using encoding.tolocal() only worked by chance in these situations: * The string is ASCII: The fast path was triggered and the string was returned unmodified. * The local encoding is UTF-8: The source and target encoding is the same. * The string is not valid UTF-8 and the native encoding is ISO-8859-1: If the string doesn’t decode using UTF-8, ISO-8859-1 is tried as a fallback. During `hg convert`, the local encoding is always UTF-8. The irony is that in this case, encoding.tolocal() behaves like what someone would expect the reverse function, encoding.fromlocal(), to do. When the locale encoding is ISO-8859-15, trying to convert a SVN repo `/tmp/a€` failed before like this: file:///tmp/a%C2%A4 does not look like a Subversion repository to libsvn version 1.14.0 The correct URL is `file:///tmp/a%E2%82%AC`. Unlike previously (with the ISO-8859-1 fallback), decoding the path using the locale encoding can fail. In this case, we have to bail out, as Subversion won’t be able to do anything useful with the path.

py3: pass URL as str Before the patch, HTTP(S) URLs were never recognized as a Subversion repository on Python 3.

convert: bail out in Subversion source if encountering non-ASCII HTTP(S) URL Before this patch, in the tested case, urllib raised `httplib.InvalidURL: URL can't contain control characters. '/\xff/!svn/ver/0/.svn' (found at least '\xff')`, which resulted in that the URL was never recognized as a Subversion repository. This patch adds a check that bails out if the URL contains non-ASCII characters. The warning is not overly user-friendly, but giving the user something to type into a search engine is definitively better than not explaining why the repository was not recognized. We could support non-ASCII chracters by quoting them before passing them to urllib. However, we would want to be compatible with what the `svn` command does, which converts the URL from the locale encoding to UTF-8, percent-encodes it and sends it to the server. If the locale encoding is not UTF-8, the behavior is IMHO not very intuitive, as the `svn` command may send different (percent-encoded) octets than what was passed on the console. Instead of copying this behavior, we better leave it forbidden.

merge with stable

compat: back out a25343d16ebe (initialize LC_CTYPE locale on all Python ...) As Yuya Nishihara pointed out, setting LC_CTYPE changes the behavior of some str methods on Python 2.

curses: back out d2227d4c9e6b (do not initialize LC_ALL to user settings) The changeset was based on a25343d16ebe, which will be backed out, too. Another fix for the problem will be resubmitted to the stable branch.

test: redirect stderr so warning messages don't change output (issue6237) clone and commit race for the lock, and if commit has to wait more than a second it prints a warning to stderr. Since this is somewhat expected here, silence it. Differential Revision: https://phab.mercurial-scm.org/D8664

locks: expect repo lock, not wlock, when writing to .hg/strip-backup/ There should be no need for a working copy lock when creating (or reading) bundles in `.hg/strip-backup/` since they don't affect the working copy. I noticed this because we have an extension that tries to strip some revisions while holding only a repo lock. I guess we have no such cases in core, which seems a bit surprising. Maybe we always take a wlock at a higher level so the working copy is not updated while the target commit is being stripped. Differential Revision: https://phab.mercurial-scm.org/D8666

graft: leverage cmdutil.check_incompatible_arguments() for --abort/--stop Differential Revision: https://phab.mercurial-scm.org/D8669

run-tests: fix escapes with conditions Before this fix, escapes with conditions in tests failed like this on Python 3: $ $PYTHON -c 'from mercurial.utils.procutil import stdout; stdout.write(b"\xff")' - \xff (no-eol) (esc) (true !) + \xff (no-eol) (esc) The unicode_escape encoding decodes br'\xff' to u'\xff'. To convert the first 256 code points to bytes with the same ordinal, the latin-1 encoding must be used. Escapes without conditions already worked before on Python 3, but not through `el == l` a few lines below the changed line in run-tests.py. I didn’t investigate further.

convert: set LC_CTYPE around calls to Subversion bindings The Subversion bindings require that LC_CTYPE is set. However, we don’t want to set it all the time, as it changes the behavior of str methods on Python 2. The taken approach is hopefully fine-grained enough to not trigger any locale-specfic behavior of the str methods and coarse-grained enough to not clutter the code. Emulating the with-statement behavior in before() and after() should be safe, as after() is always called when before() is called. hgext.convert.hg takes a similar approach.

curses: do not initialize LC_ALL to user settings (issue6358) 701341f57ceb moved the setlocale() call to right before curses was used. This didn’t fully solve the problem it was supposed to solve (locale-dependent functions, like date formatting/parsing and str methods on Python 2), but only postponed it. Initializing LC_CTYPE seems to be sufficient for curses to work correctly. Therefore LC_CTYPE is set while curses is used and reset afterwards. Some locale-dependent str methods might behave differently on Python 2 while curses is used, but that shouldn’d be a problem.

graft: leverage cmdutil.check_incompatible_arguments() for --no-commit Differential Revision: https://phab.mercurial-scm.org/D8668

graft: leverage cmdutil.check_at_most_one_arg() for --abort/--stop/--continue Differential Revision: https://phab.mercurial-scm.org/D8667

version: sort extensions by name in verbose mode External extensions can be assigned any name, but presumably most enabled extensions will be internal ones and having them sorted makes it easier to find specific ones if the list is long. The lists in `hg help extensions` are already sorted. Differential Revision: https://phab.mercurial-scm.org/D8671

crecord: stop trying to import wcurses The original import of crecord in 2008 already said "I have no idea if wcurses works with crecord...". The last reference to a Python package called wcurses is https://web.archive.org/web/20101025073658/http://adamv.com/dev/python/curses/. However, the Python package from there is called "curses" and not "wcurses". I didn’t find any evidence that it ever worked.

debian: support building a single deb for multiple py3 versions Around transitions from one python minor version to another (such as 3.7 to 3.8), the current packaging can be slightly problematic - it produces a `control` file that requires that the version of `python3` that's installed be exactly the one that was used on the build machine for the `mercurial` package, by containing a line like: Depends: sensible-utils, libc6 (>= 2.14), python3 (<< 3.8), python3 (>= 3.7~), python3:any (>= 3.5~) This is because it "knows" we only built for v3.7, which is the current default on my system. By building the native components for multiple versions, we can make it produce a line like this, which is compatible with 3.7 AND 3.8: Depends: sensible-utils, libc6 (>= 2.14), python3 (<< 3.9), python3 (>= 3.7~), python3:any (>= 3.5~) This isn't *normally* required, so I'm not making it the default. For those that receive their python3 and mercurial packages from their distro, and/or don't have to worry about a situation where the team that manages the python3 installation isn't the same as the team that manages the mercurial installation, this is probably not necessary. I chose the names `DEB_HG_*` because `DEB_*` is passed through `debuild` automatically (otherwise we'd have to explicitly allow the options through, which is a nuisance), and the `HG` part is to make it clear that this isn't a "standard" debian option that other packages might respect. Test Plan: 1. "nothing changed": - built a deb without these changes - built a deb with these changes but everything at the default - used diffoscope to compare, all differences were due to timestamps 2. "explicit is the same as implicit" (single version) - built a deb with everything at the default - built a deb with DEB_HG_PYTHON_VERSIONS=3.7 - used diffoscope to compare, all differences were due to timestamps 3. "explicit is the same as implicit" (multi version) - built a deb with DEB_HG_MULTI_VERSION=1 - built a deb with DEB_HG_PYTHON_VERSIONS=3.7 - used diffoscope to compare, all differences were due to timestamps 4. (single version, 3.7) doesn't work with python3.8 - `/usr/bin/python3.7 /usr/bin/hg debuginstall` works - `/usr/bin/python3.8 /usr/bin/hg debuginstall` crashes 5. (multi version, 3.7 + 3.8) - `/usr/bin/python3.7 /usr/bin/hg debuginstall` works - `/usr/bin/python3.8 /usr/bin/hg debuginstall` works Differential Revision: https://phab.mercurial-scm.org/D8642

merge with stable

hgweb: encode WSGI environment like OS environment Previously, the WSGI environment keys and values were encoded using latin-1. This resulted in a crash if a WSGI environment key or value could not be encoded using latin-1. On Unix, the OS environment is byte-based. Therefore we should do the reverse of what Python does for os.environ. On Windows, there’s no native byte-based OS environment. Therefore we should do the same as what mercurial.encoding does with the OS environment.

hgweb: deduplicate code A following patch will change the way keys and values are encoded. To reduce the diff, I’ve split off the uninteresting part.

curses: do not initialize LC_ALL to user settings (issue6358) 701341f57ceb moved the setlocale() call to right before curses was used. This didn’t fully solve the problem it was supposed to solve (locale-dependent functions, like date formatting/parsing), but only postponed it. Initializing LC_CTYPE seems to be sufficient for curses to work correctly. Luckily this is already done at interpreter startup on modern Python versions and, since recently, by Mercurial in the pycompat module in all other cases.

compat: initialize LC_CTYPE locale on all Python versions and platforms Previously, the LC_CTYPE locale was not initialized according to user settings on all Python versions (e.g. never on Python 2) and platforms (e.g. not on some Python < 3.8 on Windows). This broke e.g. non-ASCII filenames passed to the Subversion bindings on Python 2, resulting in error messages like "file:///tmp/a%C3%A4 does not look like a Subversion repository to libsvn version 1.14.0". The following command could be used to test this functionality. Adding it to the test suite would be pointless, as the locale is always set to "C" during test runs. @command(b'check_initial_codeset', norepo=True) def check_initial_codeset(ui): codeset1 = locale.nl_langinfo(locale.CODESET) locale.setlocale(locale.LC_ALL, '') codeset2 = locale.nl_langinfo(locale.CODESET) assert codeset1 == codeset2

merge with stable

merge: don't grab wlock when merging in memory I noticed this because we have an internal extension that does an in-memory rebase while holding only a repo lock, which resulted in a developer warning about the working copy lock being taken after the repo lock. Differential Revision: https://phab.mercurial-scm.org/D8665

pycompat: use os.fsencode() to re-encode sys.argv Historically, the previous code made sense, as Py_EncodeLocale() and fs.fsencode() could possibly use different encodings. However, this is not the case anymore for Python 3.2, which uses the locale encoding as the filesystem encoding (this is not true for later Python versions, but see below). See https://vstinner.github.io/painful-history-python-filesystem-encoding.html for a source and more background information. Using os.fsencode() is safer, as the documentation for sys.argv says that it can be used to get the original bytes. When doing further changes, the Python developers will take care that this continues to work. One concrete case where os.fsencode() is more correct is when enabling Python's UTF-8 mode. Py_DecodeLocale() will use UTF-8 in this case. Our previous code would have encoded it using the locale encoding (which might be different), whereas os.fsencode() will encode it with UTF-8. Since we don’t claim to support the UTF-8 mode, this is not really a bug and the patch can go to the default branch. It might be a good idea to not commit this to the stable branch, as it could in theory introduce regressions.

merge with stable

perf: make `hg perfwrite` more flexible The more flexible command was used recently while finding a solution for a buffering bug (eventually fixed in f9734b2d59cc (the changeset description uses a different benchmark)). In comparison to the previous version, the new version is much more flexible. While using it, the focus was on testing small writes. For this reason, by default it calls ui.write() 100 times with a single byte plus one newline byte, for 100 lines. To get the previous behavior, run `hg perfwrite --nlines=100000 --nitems=1 --item='Testing write performance' --batch-line`.

chg: fix typo

copies: implement __repr__ on branch_copies for debugging Differential Revision: https://phab.mercurial-scm.org/D8650

phases: remove useless lookup of repo[rev].rev() in _retractboundary changectx object is no longer needed thanks to a54ee130210a.

ui: replace `self._ferr` with identical `dest` Originally, it was part of a larger change that was abandoned. IMHO it makes the code slightly cleaner and saves one attribute access, so I decided to send it anyway instead of throwing it away.

absorb: make it clear what happens when no input Differential Revision: https://phab.mercurial-scm.org/D8643

tests: add hghave rule 'setprocname' to check if osutil.setprocname and use it setprocname is not present when we build a pure version. This leads to output changes in test-chg.t This should make test suite green on Python 2 pure build with chg. Differential Revision: https://phab.mercurial-scm.org/D8638

util: flush stderr explicitly after using warnings.warn() Due to some unknown reasons, when using chg with python3, the warnings.warn() output is not flushed. Fixes test-devel-warnings.t on py3 with chg.

phases: improve performance of _retractboundary The old version repeatedly converts nodes to revisions, which is a moderately expensive operation. Mapping all new changes once to revisions and back at the end reduces the time spend in _retractboundary during the unbundling of NetBSD's src from 67s to 17s. Differential Revision: https://phab.mercurial-scm.org/D8641

tests: use proctutil.stdout.write() instead of print() in test-extension.t I was debugging this test failure on python3 + chg. I get the following hunk as test failure: ``` @@ -206,6 +206,18 @@ Check normal command's load order of ext 4) bar uipopulate 5) foo reposetup 5) bar reposetup + 4) foo uipopulate (chg !) + 4) bar uipopulate (chg !) + 4) foo uipopulate (chg !) + 4) bar uipopulate (chg !) + 4) foo uipopulate (chg !) + 4) bar uipopulate (chg !) + 4) foo uipopulate (chg !) + 4) bar uipopulate (chg !) + 4) foo uipopulate (chg !) + 4) bar uipopulate (chg !) + 5) foo reposetup (chg !) + 5) bar reposetup (chg !) 0:c24b9ac61126 ``` After hours of debugging and head scracthing, I figured out that something is wrong with output flushing. I initially switched the print() statements to ui.warn() but thanks to Yuya who suggested using procutil.stdout.write() instead.

fuzz: tell manifest fuzzer about longer node hashes Differential Revision: https://phab.mercurial-scm.org/D8374

absorb: preserve changesets which were already empty Most commands in Mercurial (commit, rebase, absorb itself) don’t create empty changesets or drop them if they become empty. If there’s a changeset that’s empty, it must be a deliberate choice of the user. At least it shouldn’t be absorb’s responsibility to prune them. The fact that changesets that became empty during absorb are pruned, is unaffected by this. This case was found while writing patches which make it possible to configure absorb and rebase to not drop empty changesets. Even without having such config set, I think it’s valuable to preserve changesets which were already empty.

absorb: preserve branch-closing changesets even if empty This makes the behavior consistent with 'hg commit', which allows to create otherwise empty changesets if they close the branch. A lost branch closure can inadvertently re-open a branch, so it should be preserved.

absorb: preserve branch-changing changesets even if empty This makes the behavior consistent with 'hg commit', which allows to create otherwise empty changesets if the branch changes compared to the parent. A branch change can denote important information, so it should be preserved.

rhg: add Command trait for subcommands implemented by rhg Normalizes the interface of the cli's commands Differential Revision: https://phab.mercurial-scm.org/D8611 Differential Revision: https://phab.mercurial-scm.org/D8648

rhg: add rhg crate The goal of rhg is to speedup some of hg's commands when possible by bypassing python entirely for the time being. It is by no means a replacement for hg as it will not support extentions or configuration and implement only a subset of hg's commands and options. Only use rhg if you understand what the tradeoffs are. Differential Revision: https://phab.mercurial-scm.org/D8610

hg-core: add FindRoot operation to find repository root path Differential Revision: https://phab.mercurial-scm.org/D8609

share: provide a more useful text for hg help Differential Revision: https://phab.mercurial-scm.org/D8651

procutil: make recent fix for zombies compatible with py2 The fix in ed684a82e29b (procutil: always waiting on child processes to prevent zombies with 'hg serve', 2020-05-07) works only on Python 3 because it passes a `daemon` argument to `threading.Thread()`. Python 2 requires you to assign to the `.daemon` property instead. Python 3 also seems to support that, so this patch fixes the code by unconditionally using the old form. Differential Revision: https://phab.mercurial-scm.org/D8657

pycompat: fix crash when default locale is unknown Instead, fall back to the filesystem encoding if the default locale is unknown.

help: document meaning of '%' in graphlog output I added the feature in 14d0e89520a2 (graphlog: use '%' for other context in merge conflict, 2020-01-28), but I didn't think to look for documentation until today (because an internal user asked about it). Differential Revision: https://phab.mercurial-scm.org/D8649

py3: fix broken man page generation, it was generating `(default: NUL*)` `bytes(default)` was producing things like `(default: \x00)` when handed non-bytes values such as `1`, `10`, or `True`. The man page generation would apparently ignore these bytes and produce man pages that had the string `(default: )`. Test Plan: - Ran `cd doc; python3 gendoc.py "hg.1.gendoc"` and grepped for bad output - Ran `make deb`, extracted the deb, manually inspected `hg.1` file. Differential Revision: https://phab.mercurial-scm.org/D8639

hg-core: add Operation interface for high-level hg operations A distinction is made between operations and commands. An operation is a high-level function of mercurial whereas a command is what is exposed by the cli. A single command can use several operations to achieve its goal. Differential Revision: https://phab.mercurial-scm.org/D8608

rust: do a clippy pass This is the result of running `cargo clippy` on hg-core/hg-cpython and fixing the lints that do not require too much code churn (and would warrant a separate commit/complete refactor) and only come from our code (a lot of warnings in hg-cpython come from `rust-cpython`). Most of those were good lints, two of them was the linter not being smart enough (or compiler to get up to `clippy`'s level depending on how you see it). Maybe in the future we could have `clippy` be part of the CI. Differential Revision: https://phab.mercurial-scm.org/D8635

py3: fix comparison between int and None If stop is None, the condition was always false on Python 2, as None compares smaller than ints. Therefore we make the condition false if stop is None.

py3: pass regex as bytes

py3: avoid using %r format on bytes Before the patch, the 'b' prefix appeared in the formatted string. Wrapping the bytes as pycompat.bytestr solves this problem. Eventually, I think that we should move away from using %r (like 975e517451a6 and 4d6019c0e0ef did), but that would change output of non-ASCII bytes on Python 2, so we can’t do it on the stable branch. Also, many places continue to use %r, so it would be a good idea to do the change all at once.

py3: use `%d` for int in % formatting On Python 3, `%s` is an alias to `%b`, which requires that the object implements `__bytes__()`, which is not the case for `int`.

py3: pass native string to urlreq.url2pathname() Of course, I’m not happy with the warning, but it’s better than crashing. Solving the problem properly is hard, and non-UTF-8 percent-encoded bytes in file URLs seem rare enough to block solving that all file URLs (even if not SVN-specific) will cause a crash.

py3: suppress DeprecationWarning about deprecated base64 module aliases base64.encodestring() / base64.decodestring() were renamed to base64.encodebytes() / base64.decodebytes() in Python 3. The old names still worked, but raised a DeprecationWarning.

py3: use `pycompat.ziplist()`