push-dests: rework the handling of default value This new core is more straightforward and doing this early will make the next changeset simpler. Differential Revision: https://phab.mercurial-scm.org/D10384

outgoing: merge the code handling --graph with the main one The --graph code had its own copy of the logic. With the previous reorganisation of the code, we can now merge it with the main code, reducing fragile complication. As a side effect, `hg out --graph` now use the right return code when they are nothing outgoing. This explain the change to output in `tests/test-largefiles-misc.t`. Differential Revision: https://phab.mercurial-scm.org/D10383

outgoing: move filtering logic in its own function This move code dedicated to a single purpose together and make the main code simpler. Right when we are getting ready to make it more complex :-D Differential Revision: https://phab.mercurial-scm.org/D10382

outgoing: make `recurse` a real function If we want to use this in a loop, we need to be able to pass argument. Differential Revision: https://phab.mercurial-scm.org/D10381

outgoing: remove some early return Since 066b8d8f75b8, the push command accept multiple destination. However `hg outgoing` does not. On the way to fix this, we need to clean up the outgoing code. We start with removing some early return to make the code ready to house more changes. Differential Revision: https://phab.mercurial-scm.org/D10380

infinitepush: use the new function to determine push destination Since 066b8d8f75b8, the push command accept multiple destination. `infinitepush` was not aware of that. We now use the new `urlutil.get_push_paths` function to determine the push destination, fixing the issue. This will also make future evolution of that logic transparent for infinitepush We still disallow push to multiple destinations if infinite push is enabled because I don't know what this means for infinite push. However user will now get a clear error message instead of a crash. Differential Revision: https://phab.mercurial-scm.org/D10379

urlutil: add a `get_pull_paths` to perform the pull destination logic As is this changeset does not change anything. However having an official empty point will help unifying the logic and encapsulate the details and update the logic to support path definition pointing to multiple other path. Differential Revision: https://phab.mercurial-scm.org/D10378

urlutil: add a `get_push_paths` to perform the push destination logic As is this changeset does not change anything. However having an official empty point will help unifying the logic and encapsulate the details and update the logic to support path definition pointing to multiple other path. Differential Revision: https://phab.mercurial-scm.org/D10377

urlutil: extract `parseurl` from `hg` into the new module The new module is well fitting for this new code. And this will be useful to make the gathered code collaborate more later. Differential Revision: https://phab.mercurial-scm.org/D10375

urlutil: extract `url` related code from `util` into the new module The new module is well fitting for this new code. And this will be useful to make the gathered code collaborate more later. Differential Revision: https://phab.mercurial-scm.org/D10374

urlutil: extract `path` related code into a new module They are a lot of code related to url and path handling scattering into various large module. To consolidate the code before doing more change (for defining "multi-path"), we gather it together. Differential Revision: https://phab.mercurial-scm.org/D10373

help: document the `path://` url scheme If we want people to use it, we need to document it. Differential Revision: https://phab.mercurial-scm.org/D10372

help: point to `hg help urls` in `hg help config.paths` This seems useful to point at what people can put as value for these config. Differential Revision: https://phab.mercurial-scm.org/D10371

merge with stable

narrow: add capabilities for local repos, not just remote peers This fixes the bug where running `hg clone --narrow ./local-repo` fails with abort: server does not support narrow clones even when the server has narrow enabled. Differential Revision: https://phab.mercurial-scm.org/D10357

typing: make minor adjustments to mercurial/util.py to pass pytype checking I'm assuming the wrong-arg-count is a pytype bug, because this code is used by the config object. Avoiding initializing `_lrucachenode` node points to None eliminates a few `is not None` assertions, but apparently not all of them. I can't figure out why it gets confused over the state where these new assertions are. Differential Revision: https://phab.mercurial-scm.org/D10276

util: fix the signature for the pypy override of sortdict.update() PyCharm flagged this as not matching the base class signature. Not sure if there was anything supplying these extra arguments though. Differential Revision: https://phab.mercurial-scm.org/D10275

typing: add type hints to mercurial/error.py The only slightly unusual things here are that `location` is passed to `ParseError` and both bytes and an int (so this accepts both), and the message passed `ProgrammingError` is immediately converted to str. Therefore it is typed as `AnyStr`, because there are a couple of instances that are already passed as str. There are a couple of places where bytes are being passed to builtin exceptions that might need to be converted to str. Differential Revision: https://phab.mercurial-scm.org/D10274

tests: add a (very slow) test that executes pytype This is an updated form of D7295, and completes successfully with pytype 2021.03.22. The 5 or so crashes that were mostly in the hgweb files seems to have been fixed in 2021.03.10. Differential Revision: https://phab.mercurial-scm.org/D10237

upgrade: do not hardcore file extension of revlogs This logic already lives inside the `store` module. So lets reuse it instead. Differential Revision: https://phab.mercurial-scm.org/D10317

upgrade: take advantage of the new information returned by `store.walk` Before this change the upgrade code had to analyse filename to process them directly. Lets keep that logic private to the store and more to a more robust explicit approach. Differential Revision: https://phab.mercurial-scm.org/D10316

store: also return some information about the type of file `walk` found We start returning of 4th information in the `store.walk` return tuple: the type of the file. This will make it easier for caller to determine which kind of file they are looking at. This should especically help with the `upgrade-repo` code that has to do a lot of fragile index's file name comparison. Differential Revision: https://phab.mercurial-scm.org/D10315

tests: skip test-git-interop.t on Windows Casefolding isn't handled in dirstate yet, triggering a bunch of assertions. But while this is more correctly `no-icasefs`, it's more likely to get attention if someone sees it. I'd just rather not have it adding to the noise on Windows for now. Differential Revision: https://phab.mercurial-scm.org/D10312

contrib: restore the `hg fix` configuration in the examples After decc3bd3f20d, running `black` will DTRT, but running `hg fix` did nothing (unless the example config file was %included, in which case it truncated the file instead of formatting it). I'm not sure why that was happening, but let's not leave a code shredder laying around. Differential Revision: https://phab.mercurial-scm.org/D10311

blackbox: fix type error on log rotation on read-only filesystem Grepping around, the code uses either encoding.strtolocal or stringutil.forcebytestr in this situation. No idea which is best. Differential Revision: https://phab.mercurial-scm.org/D10293

rust: Remove use of `py.eval()` The previous Rust code allocated an intermediate `Vec`, converted that to a Python list, then used `eval` to run Python code that converts that list to a Python set. rust-cpython exposes Rust bindings for Python sets, let’s use that instead to construct a set directly. Differential Revision: https://phab.mercurial-scm.org/D10328

rust: Remove the compile-time 'dirstate-tree' feature flag This code has compiler errors since it is not built on CI and nobody has been working on it for some time. We (Octobus) are still pursuing status optimizations based on a tree data structure for the dirstate, but upcoming patches will use a run-time opt-in instead of compile-time, so that at least corresponding Rust code keeps compiling when other changes are made. Differential Revision: https://phab.mercurial-scm.org/D10329

procutil: avoid using os.fork() to implement runbgcommand We ran into the following deadlock: - some command creates an ssh peer, then raises without explicitly closing the peer (hg id + extension in our case) - dispatch catches the exception, calls ui.log('commandfinish', ..) (the sshpeer is still not closed), which calls logtoprocess, which calls procutil.runbgcommand. - in the child of runbgcommand's fork(), between the fork and the exec, the opening of file descriptors triggers a gc which runs the destructor for sshpeer, which waits on ssh's stderr being closed, which never happens since ssh's stderr is held open by the parent of the fork where said destructor hasn't run Remotefilelog appears to have a hack around this deadlock as well. I don't know if there's more subtlety to it, because even though the problem is determistic, it is very fragile, so I didn't manage to reduce it. I can imagine three ways of tackling this problem: 1. don't run any python between fork and exec in runbgcommand 2. make the finalizer harmless after the fork 3. close the peer without relying on gc behavior This commit goes with 1, as forking without exec'ing is tricky in general in a language with gc finalizers. And maybe it's better in the presence of rust threads. A future commit will try 2 or 3. Performance wise: at low memory usage, it's an improvement. At higher memory usage, it's about 2x faster than before when ensurestart=True, but 2x slower when ensurestart=False. Not sure if that matters. The reason for that last bit is that the subprocess.Popen always waits for the execve to finish, and at high memory usage, execve is slow because it deallocates the large page table. Numbers and script: before after mem=1.0GB, ensurestart=True 52.1ms 26.0ms mem=1.0GB, ensurestart=False 14.7ms 26.0ms mem=0.5GB, ensurestart=True 23.2ms 11.2ms mem=0.5GB, ensurestart=False 6.2ms 11.3ms mem=0.2GB, ensurestart=True 15.7ms 7.4ms mem=0.2GB, ensurestart=False 4.3ms 8.1ms mem=0.0GB, ensurestart=True 2.3ms 0.7ms mem=0.0GB, ensurestart=False 0.8ms 0.8ms import time for memsize in [1_000_000_000, 500_000_000, 250_000_000, 0]: mem = 'a' * memsize for ensurestart in [True, False]: now = time.time() n = 100 for i in range(n): procutil.runbgcommand([b'true'], {}, ensurestart=ensurestart) after = time.time() ms = (after - now) / float(n) * 1000 print(f'mem={memsize / 1e9:.1f}GB, ensurestart={ensurestart} -> {ms:.1f}ms') Differential Revision: https://phab.mercurial-scm.org/D9019

share: store relative share paths with '/' separators I created a relative share in Windows and tried to use it in WSL, and it failed: abort: .hg/sharedpath points to nonexistent directory /mnt/c/Users/Matt/hg-review/.hg/..\..\hg\.hg Use `normpath` on the read side so that the code has the usual Windows style paths it always had (I don't think that matters much), but it also eliminates the directory escaping path components in the case where the path is printed. This will not fix repositories that have already been created, but it's trivial enough to hand edit the file to correct it. Differential Revision: https://phab.mercurial-scm.org/D10330

unit-tests: Fix `cargo test` on 32-bit platforms Fixes https://bz.mercurial-scm.org/show_bug.cgi?id=6506 This makes `IndexEntryBuilder::build`, which is only used in unit tests, use `u32` or `u64` instead of platform-dependent `usize` when packing binary data to be used at test input. To run Rust unit tests in 32-bit mode in a x86-64 environment, use: rustup target add i686-unknown-linux-gnu # Once (cd rust && cargo test --target i686-unknown-linux-gnu) Differential Revision: https://phab.mercurial-scm.org/D10351

rename: add --forget option and stop suggesting `hg revert` for undoing Differential Revision: https://phab.mercurial-scm.org/D10355

win32: enable legacy I/O mode to fix missing pager output on Windows with py3 The equivalent interpreter option is set by wrapper.exe, but this *.bat file is what gets installed in a venv. Without this mode, any command that spins up a pager has no output, unless the pager is explicitly disabled. The variable is set inside the `setlocal` scope to keep it from leaking into the environment after the bat file exits. We should probably still figure out how to ship a compiled hg.exe when installing with `pip`, because the binary does other things like enable long filename support. But this avoids the dangerous and confusing lack of output in the meantime. Differential Revision: https://phab.mercurial-scm.org/D10354

persistent-nodemap: enable the feature by default when using Rust As discussed at the 5.6 sprint, we can make it enabled by default, but only for Rust installation. Differential Revision: https://phab.mercurial-scm.org/D9765

persistent-nodemap: disable it for test-wireproto-content-redirects.t This is not relevant for the test and this will avoid a lot of variations. Differential Revision: https://phab.mercurial-scm.org/D10338

persistent-nodemap: disable it for test-wireproto-command-capabilities.t This is not relevant for the test and this will avoid a lot of variations. Differential Revision: https://phab.mercurial-scm.org/D10337

persistent-nodemap: disable it unconditionally for test-wireproto-caching.t This is not relevant for the test and this will avoid a lot of variations. Differential Revision: https://phab.mercurial-scm.org/D10336

persistent-nodemap: disable it unconditionally for test-ssh-proto-unbundle.t This is not relevant for the test and this will avoid a lot of variations. Differential Revision: https://phab.mercurial-scm.org/D10335

persistent-nodemap: disable it unconditionally for test-ssh-proto.t This is not relevant for the test and this will avoid a lot of variations. Differential Revision: https://phab.mercurial-scm.org/D10334

persistent-nodemap: disable it unconditionally for test-share-safe.t This is not relevant for the test and this will avoid a lot of variations. Differential Revision: https://phab.mercurial-scm.org/D10333

persistent-nodemap: disable it unconditionally for test-http-protocol.t This is not relevant for the test and this will avoid a lot of variations. Differential Revision: https://phab.mercurial-scm.org/D10332

persistent-nodemap: disable it unconditionally for test-http-bad-server.t This is not relevant for the test and this will avoid a lot of variations Differential Revision: https://phab.mercurial-scm.org/D10331

clang-format: run the formatter on mercurial/cext/revlog.c This fix `test-check-clang-format.t` that has been complaining for a while. Differential Revision: https://phab.mercurial-scm.org/D10327

revlog-compression: use zstd by default (if available) As see in changeset bb271ec2fbfb, zstd is 20% to 50% faster for reading and writing. Use take advantage of the new config behavior to try zstd by default, falling back to zlib is zstd is not available on that plateform. Differential Revision: https://phab.mercurial-scm.org/D10326

rhg: make rhg recognise it supports zstd compression for revlogs It already did, but was not aware of it. Differential Revision: https://phab.mercurial-scm.org/D10324

match: convert O(n) to O(log n) in exactmatcher.visitchildrenset When using narrow, during rebase this is called (at least) once per directory in the set of files in the commit being rebased. Every time it's called, we did the set arithmetic (now extracted and cached), which was probably pretty cheap but not necessary to repeat each time, looped over every item in the matcher and kept things that started with the directory we were querying. With very large narrowspecs, and a commit that touched a file in a large number of directories, this was slow. In a pathological repo, the rebase of a single commit (that touched over 17k files, I believe in approximately as many directories) with a narrowspec that had >32k entries took 8,246s of profiled time, with 5,007s of that spent in visitchildrenset (transitively). With this change, the time spent in visitchildrenset is less than 34s (which is where my profile cut off). Most of the remaining time was network access due to our custom remotefilelog-based setup not properly prefetching. Differential Revision: https://phab.mercurial-scm.org/D10294

exthelper: improve docs to indicate what module vars are needed I recently tried creating an extension "from scratch" using exthelper, and it wasn't obvious that you needed these. I believe that a careful reading of one of the comments would tell you that they were required, but it's easy to miss and having the examples be "complete" is helpful. Differential Revision: https://phab.mercurial-scm.org/D10295

tests: restore the ability to run `black` on Windows The hghave test for black silently stopped working with the change in 08fd76a553c9. This was the output of what it was hitting when run in the test environment: Traceback (most recent call last):\r (esc) File "c:\\users\\matt\\appdata\\local\\programs\\python\\python39\\lib\\runpy.py", line 197, in _run_module_as_main\r (esc) return _run_code(code, main_globals, None,\r (esc) File "c:\\users\\matt\\appdata\\local\\programs\\python\\python39\\lib\\runpy.py", line 87, in _run_code\r (esc) exec(code, run_globals)\r (esc) File "c:\\Users\\Matt\\AppData\\Roaming\\Python\\Python39\\Scripts\\black.exe\\__main__.py", line 4, in <module>\r (esc) File "C:\\Users\\Matt\\AppData\\Roaming\\Python\\Python39\\site-packages\\black\\__init__.py", line 70, in <module>\r (esc) CACHE_DIR = Path(user_cache_dir("black", version=__version__))\r (esc) File "C:\\Users\\Matt\\AppData\\Roaming\\Python\\Python39\\site-packages\\appdirs.py", line 293, in user_cache_dir\r (esc) path = os.path.normpath(_get_win_folder("CSIDL_LOCAL_APPDATA"))\r (esc) File "C:\\Users\\Matt\\AppData\\Roaming\\Python\\Python39\\site-packages\\appdirs.py", line 481, in _get_win_folder_with_pywin32\r (esc) dir = shell.SHGetFolderPath(0, getattr(shellcon, csidl_name), 0, 0)\r (esc) pywintypes.com_error: (-2147024893, '$ENOTDIR$.', None, None)\r (esc) [1] Differential Revision: https://phab.mercurial-scm.org/D10310

setup: copy python3.dll next to hg.exe when building on Windows for hgext.git I thought I took care of this already, but it must have been that I just manually copied the file over locally when debugging why the pygit2 library wasn't loading. The problem with that is what was copied over was from py38, and then running a py39 build hard crashed when the extension was loaded. Differential Revision: https://phab.mercurial-scm.org/D10301

tests: stablize test-hook.t on Windows Apparently, hooks can't run `echo` directly, even from MSYS. Differential Revision: https://phab.mercurial-scm.org/D10300

tests: handle Windows file separator differences in test-config.t Differential Revision: https://phab.mercurial-scm.org/D10299

tests: update the detailed exit codes for icasefs gated tests The fact that there's already a detailed exit code after the last change here in test-casecollision-merge.t gives me some pause, but maybe it was found and changed manually? Differential Revision: https://phab.mercurial-scm.org/D10298

tests: update the detailed exit codes in test-phabricator.t I'm guessing this has slipped through because most people don't have the necessary `pytest-vcr` package installed. Differential Revision: https://phab.mercurial-scm.org/D10297

tests: avoid using rebaseskipobsolete=0 I'm about to delete the `rebaseskipobsolete` config. This patch updates a test to get hidden commits into the rebase state by using `hg debugobsolete` instead of setting `rebaseskipobsolete=0`. Differential Revision: https://phab.mercurial-scm.org/D10270

rebase: when using --keep, don't care about pruned commits or divergence `hg rebase --keep` creates duplicate commits (not successors), so I was surprised that it still skips pruned commits and errors out if it "would cause divergence" (it wouldn't). I guess this was just an oversight. We didn't have any tests for it, so I also included that. Differential Revision: https://phab.mercurial-scm.org/D10269

rebase: remove duplicate initialization of a field `obsolete_with_successor_in_destination` is already initialized in the constructor. Differential Revision: https://phab.mercurial-scm.org/D10268

rust: bump rust-cpython version to 0.5.2 we need a newer version to define "property" on Rust defined object. Differential Revision: https://phab.mercurial-scm.org/D10308

revlog: directly use the Struct object for related operation The Struct object has all the piece we needs, so no need to duplicate information on the revlog itself. Differential Revision: https://phab.mercurial-scm.org/D10307

revlog: move the "index header" struct inside revlog.utils.constants The struct was previous called "version", but this is actually "version" + "flags". So header seems like a better name. The move to the `constants` module has the same motivation as the INDEX_ENTRY_V# ones. Differential Revision: https://phab.mercurial-scm.org/D10306

revlog: move the details of revlog "v2" index inside revlog.utils.constants the revlog module is quite large and this kind of format information would handy for other module. So let us start to gather this information about the format in a more appropriate place. We update various reference to this information to use the new "source of truth" in the process. Differential Revision: https://phab.mercurial-scm.org/D10305

revlog: move the details of revlog "v1" index inside revlog.utils.constants The revlog module is quite large and this kind of format information would handy for other module. So let us start to gather this information about the format in a more appropriate place. We update various reference to this information to use the new "source of truth" in the process. Differential Revision: https://phab.mercurial-scm.org/D10304

revlog: move the details of revlog "v0" index inside revlog.utils.constants the revlog module is quite large and this kind of format information would handy for other module. So let us start to gather this information about the format in a more appropriate place. Differential Revision: https://phab.mercurial-scm.org/D10303

revlog: add some comment in the header sections We are about to add more content so let us organise the existing content first. Differential Revision: https://phab.mercurial-scm.org/D10302

store: drop the `filefilter` argument to `_walk` No code use it anywhere. Dropping it will help replacing the function with something with a more precise semantic. Differential Revision: https://phab.mercurial-scm.org/D10314

store: document the `walk` method Differential Revision: https://phab.mercurial-scm.org/D10313

revlog: fix error about unknown compression format in py3 In py2, the error is something like: abort: unknown compression type 'x'! In py3, we get the following unhelpful message: abort: unknown compression type <memory at 0x7f4650b5cdc8>! Switch to something like: abort: unknown compression type 78! Differential Revision: https://phab.mercurial-scm.org/D10318

revlog-compression: fix computation of engine availability We don't just need the engine to be define, we need it to be available and able to do be used for revlog compression. Without this change, `zstd` could be selected as a viable option for repository creation on platform where it is not available. Differential Revision: https://phab.mercurial-scm.org/D10325

test: explicitly use zlib compression in tests/test-share-safe.t We need the implicit value to be explicit until we can change the default in some case. Differential Revision: https://phab.mercurial-scm.org/D10323

test: explicitly use zlib compression in tests/test-upgrade-repo.t We need the implicit value to be explicit until we can change the default in some case. Differential Revision: https://phab.mercurial-scm.org/D10322

test: explicitly use zlib compression in tests/test-repo-compengines.t We need the implicit value to be explicit until we can change the default in some case. Differential Revision: https://phab.mercurial-scm.org/D10321

remotefilelog: include file contents in bundles produced during strip `hg strip` and other things that use repair.strip (such as the narrow extension's `hg tracked --removeinclude`) will "save" some commits that have a higher revision number than the oldest commit we're stripping, but aren't actually descended from any of the commits that we're stripping. It saves them in a bundle, and then reapplies them to the repo. Remotefilelog doesn't generally participate in strip, it doesn't contribute files to either the backup bundle or the "saved" bundle, and doesn't adjust linknodes when commits are stripped. This can break things like push, which rely on the linknodes. This change makes it so that remotefilelog includes files in these bundles during strip operations. During reapplication, the files are reapplied from the bundle, and the linknode is properly updated. Differential Revision: https://phab.mercurial-scm.org/D10320

tests: add test-remotefilelog-strip.t to demonstrate an issue with linknodes ### Background Every time a commit is modified, remotefilelog updates the metadata for the file object to point to the new commit (I believe that this is different from non-remotefilelog hg, which leaves the linkrevs pointing to the obsolete commits; doing otherwise would involve changing data in the middle of revlogs). With `hg strip` (or other things that use repair.strip()), when you strip a commit that's not the tip of the revlog, there may be commits after it in revnum order that aren't descended from it and don't need to be (and shouldn't be) stripped. These are "saved" by strip in a bundle, and that bundle is reapplied after truncating the relevant revlogs. ### The problem Remotefilelog generally avoids being involved at all in strip. Currently, that includes even providing file contents to this backup bundle. This can cause the linknode to point to a changeset that is no longer in the repository. Example: ``` @ 3 df91f74b871e | | x 2 70494d7ec5ef |/ | x 1 1e423846dde0 |/ o 0 b292c1e3311f ``` Commits 1, 2, and 3 are related via obsolescence, and are description-only changes. The linknode for the file in these commits changed each time we updated the description, so it's currently df91f7. If I strip commits 1 and 3, however, the linknode *remains* df91f7, which no longer exists in the repository. Commit 70494d was "saved", stripped, and then reapplied, so it is in the repository (as revision 1 instead of 2 now), and was unobsoleted since the obsmarker was stripped as well. The linknode for the file should point to 70494d, the most recent commit that is in the repository that modified the file. Remotefilelog has some logic to handle broken linknodes, but it can be slow. We have actually disabled it internally because it's too slow for our purposes. Differential Revision: https://phab.mercurial-scm.org/D10319

mergestate: remove unused import Differential Revision: https://phab.mercurial-scm.org/D10291

deb: avoid use of [[ in 'rules' file It's not supported by posix shell, and apparently my build system uses that. Differential Revision: https://phab.mercurial-scm.org/D10292

refactor: prefer checks against nullrev over nullid A common pattern is using a changeset context and obtaining the node to compare against nullid. Change this to obtain the nullrev instead. In the future, the nullid becomes a property of the repository and is no longer a global constant, so using nullrev is much easier to reason about. Python function call overhead makes the difference moot, but future changes will result in more dictionary lookups otherwise, so prefer the simpler pattern. Differential Revision: https://phab.mercurial-scm.org/D10290

refactor: prefer lookup by revision, even for null While the nullid lookup is a special case, it is still more complicated. The common pattern is to lookup via nullrev so be consistent here. Differential Revision: https://phab.mercurial-scm.org/D10280

setdiscovery: simplify by using tiprev directly tip() uses tiprev() and reads the node from it, so drop a layer of indirection. Differential Revision: https://phab.mercurial-scm.org/D10289

test: enforce master to be the default branch in test Newer git issue a message about this. Differential Revision: https://phab.mercurial-scm.org/D10281

fix: merge imports Differential Revision: https://phab.mercurial-scm.org/D10277

tests: update divergence test for `hg fix` to actually result in divergence We have a test that checks that `hg fix` errors out if it might cause divergence. However, the test simply prunes the commit it then tries to fix, so fixing it wouldn't actually cause divergence. That works because the implementation is simple enough that it doesn't notice the difference. I'm about to make the implementation smarter, so let's fix the test first. Differential Revision: https://phab.mercurial-scm.org/D10267

rebase: don't call rewriteutil.precheck() with to-be-skipped commits It's clearly incorrect to call `rewriteutil.precheck()` for commits that we're not about to rewrite. We haven't noticed yet because the function doesn't check for divergence, but I'm about to teach it to do that. Differential Revision: https://phab.mercurial-scm.org/D10259

reabase: call rewriteutil.precheck() a bit later We now filter out descendants of divergence-causing commits in `_handleskippingobsolete()`. The filtered-out commits are removed from the rebase set (`destmap` and `state`). We should therefore call `rewriteutil.precheck()` after `_handleskippingobsolete()`. This patch does that. It hasn't mattered so far because `rewriteutil.precheck()` doesn't yet check for divergence, but it will soon. This affects one test where we now fail because the user is trying to rebase an ancestor instead of failing because they tried to rebase a public commit. We have several similar tests just after, where we still fail because of the phase, so that seems fine. The difference in behavior also seems fine to me. Differential Revision: https://phab.mercurial-scm.org/D10258

rebase: set `prepared = True` at very end of `_preparenewrebase()` Once we've set `rebaseruntime.prepared = True`, `rebaseruntime.repo` starts returning the unfiltered repo. That will make my next patch break, because that patch moves the call to `rewriteutil.precheck()` after the call to `_handleskippingobsolete()`, which current happens after `prepared = True`. We therefore need to prepare by moving `prepared = True` a bit later, after `_handleskippingobsolete()`. I don't think that matters for that call. Differential Revision: https://phab.mercurial-scm.org/D10257

rebase: filter out descendants of divergence-causing commits earlier `hg rebase` treats obsolete commits differently depending what has happened to the commit: 1) Obsolete commit without non-obsolete successors: Skipped, and a note is printed ("it has no successor"). 2) Obsolete commit with a successor in the destination (ancestor of it): Skipped, and a note is printed ("already in destination"). 3) Obsolete commit with a successor in the rebase set: The commit and its descendants are skipped, and a note is printed ("not rebasing <commit> and its descendants as this would cause divergence"), unless `allowdivergence` config set. 4) Obsolete commit with a successor elsewhere: Error ("this rebase will cause divergences"), unless `allowdivergence` config set. Before this patch, we did all those checks up front, except for (3), which was checked later. The later check consisted of two parts: 1) filtering out of descendants, and 2) conditionally printing message if the `allowdivergence` config was not set. This patch makes it so we do the filtering early. A consequence of filtering out divergence-causing commits earlier is that we rebase commits in slightly different order, which has some impact on tests. Differential Revision: https://phab.mercurial-scm.org/D10249

rebase: clarify names of variables and function related to obsolete revisions Differential Revision: https://phab.mercurial-scm.org/D10248

rebase: calculate obsolescense-related info earlier My goal is to use `rewriteutil.precheck()` in the rebase code. Since rebase does its own handling of divergent commits (it skips them instead of erroring out), we need to have divergence-causing commits filtered out early. This patch helps prepare for that. Differential Revision: https://phab.mercurial-scm.org/D10247

rebase: let _handleskippingobsolete(self) read directly from self The function already has `self` as an argument, so there's no need to pass data from `self` into it. Differential Revision: https://phab.mercurial-scm.org/D10246

tests: test divergence created during interrupted rebase If a rebase runs into conflicts and the user somehow rewrites an unrebased commit in the rebase set while the rebase is interrupted, continuing it might result in divergence. It turns out that we decide to skip the commit. That seems to make sense, but it wasn't obvious to me that that's what we should do. Either way, this patch adds a test case for the current behavior. Differential Revision: https://phab.mercurial-scm.org/D10256

tests: split up test-rebase-obsolete.t in four pieces The test case took 42 seconds to run the test before this patch and 12 seconds after (wall time, of course). Differential Revision: https://phab.mercurial-scm.org/D10255

path: error out if the `path://` reference point to an unknown path Differential Revision: https://phab.mercurial-scm.org/D10265

path: forbid chaining `path://` definitions To have `path://xxx` referencing paths that use `path://` too, we need to analyze dependencies to initialize them in the right order (and to detect cycle). I don't want to deal with that right now, so I just disallow it for now. Differential Revision: https://phab.mercurial-scm.org/D10264

path: introduce a `path://` syntax to reference other path name This make it easier for a path to reuse the same location of another path with different parameter. This is useful to create path "alias" with common config option. This will become very useful to create path that reference a list of other path. This changeset focus on implemented the basic feature, future changesets will deal with various error management (and associated testing). Differential Revision: https://phab.mercurial-scm.org/D10263

path: extract sub-option logic into its own method We will need to re-use this logic for `path://` so we first extract it into its own method. Differential Revision: https://phab.mercurial-scm.org/D10262

path: extract the path validation logic into its own submethod We will need to re-use this logic for `path://` so we first extract it into its own method. Differential Revision: https://phab.mercurial-scm.org/D10261

path: move handling of "default" (*) suboptions value inside __init__ With the introduction of `path://` scheme the handling of default value will need to be subtler. We do simple code movement first to clarify the future changes. Differential Revision: https://phab.mercurial-scm.org/D10260

rhg: Initial support for the 'status' command Only comparing the working directory with its first parent revision is supported. The core logic of dirstate handling and `stat`’ing files was already in `hg-core` supporting Python-based hg with Rust extensions, so this is mostly plumbing to rhg’s CLI. For now the command is experimental and disabled by default, since it has some bugs that causes a number of tests to fail. These failures can be seen with: tests/run-tests.py --rhg --extra-config-opt rhg.status=true Differential Revision: https://phab.mercurial-scm.org/D10239

rhg: raise wdir specific error for `hg debugdata` Helps remove the conditional in `test-debugcommands.t` for rhg. Differential Revision: https://phab.mercurial-scm.org/D10254

rhg: add support for detailed exit code for ConfigParseError This patch adds basic support for detailed exit code to rhg with support for ConfigParseError. For now, if parsing the config results in error, we silently fallbacks to `false`. The python version in this case emits a traceback. Differential Revision: https://phab.mercurial-scm.org/D10253

contributor: change mentions of mpm to olivia Matt Mackall is now Olivia Mackall. I reached out to her about changing the copyright notices to reflect this change and she gave me the green light, so I changed everything relevant. Differential Revision: https://phab.mercurial-scm.org/D10266

fuzz: use Python 3 in makefile As in the previous patch, I copied the snippet defining `PYTHON` from the root Makefile. Differential Revision: https://phab.mercurial-scm.org/D10252

makefile: use Python 3 by default when building docs as well I copied the snippet defining `PYTHON` from the root Makefile. Differential Revision: https://phab.mercurial-scm.org/D10251

tests: replace some `python` by `"$PYTHON"` to not only use Python 2 I deleted my `python` binary as a test and a few tests started failing. Differential Revision: https://phab.mercurial-scm.org/D10250

copies: distinguish between merged and touched files during upgrade This a filenode being different between p1, p2 and result does not necessarily means a merges happens. For example p2 could be a strict newer version of p1, so the p2 version is picked by the manifest merging, but then the file is manually updated before the commit. In this case the file should be detected as touched. Differential Revision: https://phab.mercurial-scm.org/D10220

copies: detect files as `touched/salvaged` if they only existed on one side The file cannot be merged if there was content to merge on the other side. So the previous record was wrong. In the general case, the file existed only on one side and got touched during the merge. So it should detected as touched. They are a special case where the merge manually prevent the file to be deleted. In this case the file is marked as `salvaged`. The result of this `salvaged` recording, copy-tracing-wise, is the same as recording it as `merged`. This is probably why they were recorded as `merged` in the first place. Differential Revision: https://phab.mercurial-scm.org/D10219

test-copies: add a test updating file content while salvaging it A deleted file is brought back during a merge. Its content is changed in the same go. This reveal some issue with the upgrade code. Differential Revision: https://phab.mercurial-scm.org/D10088

test-copies: show some wrong ChangedFiles upgrade For file present only on one side and touched during merge, the upgrade code confused them as "merged". However they should be either "touched", or "salvaged" but they are currently recorded as "merged". See the next changesets for more details on these cases and fix. Differential Revision: https://phab.mercurial-scm.org/D10218

commit: get info from mergestate whether a file was merged or not While commiting a merge, the commit code does not know whether a file was merged during `hg merge` or not. This leads the commit code to look for filelog ancestor to choose parents of new filelog created on merge commit. This leads to wrong results in some cases as demonstrated by previous patch. From this patch, we start storing information about merged files in mergestate in stateextras and then use that on commit to detect whether we need to set two parents or not. Differential Revision: https://phab.mercurial-scm.org/D10149

typing: disable processing of mercurial/upgrade_utils/actions.py There was a recent regression in pytype that causes processing to stall and consume memory, and it doesn't sound like it will be fixed any time soon. Differential Revision: https://phab.mercurial-scm.org/D10273

revset: introduce a `nodefromfile` revset I though we had one, but actually we don't seem to. So here is a revset to reuse a list of node previously stored. Differential Revision: https://phab.mercurial-scm.org/D10230

commands: necessary annotations and assertions to pass pytype This is a slightly less forceful incarnation of D7384, where pytype can be appeased with some assertions rather than disabling warnings. Differential Revision: https://phab.mercurial-scm.org/D10236

typing: mark the argument to mercurial.i18n.gettext() non-Optional Few if any of the callers are handling a `None` return, which is making pytype complain. I tried adding @overload definitions to indicate the bytes -> bytes and None -> None relationship, but pytype doesn't seem to apply that to `_()` through the function assignment. What did work was to change `_()` into its own function that called `gettext()`, but that has an extra function call overhead. Even putting that function into an `if pycompat.TYPE_CHECKING` block and leaving the existing assignments in the `else` block caused pytype to lose track of the @overloads. At that point, I simply gave up. PyCharm doesn't like that it can return None, given the new type hints, but pytype doesn't complain about this nor does it see any callers passing None. The most important thing here is to catch str being passed anyway. Differential Revision: https://phab.mercurial-scm.org/D10235

revlog: convert an Abort message to bytes Flagged by pytype. Differential Revision: https://phab.mercurial-scm.org/D10234

setdiscovery: add a discovery.grow-sample.dynamic option The current discovery dynamically adapt to complex situations. This makes it quick and effective, but also harder so study the effects of other improvements in such complex situation. So we add a new option to control this too. Differential Revision: https://phab.mercurial-scm.org/D10233

setdiscovery: rearrange code deciding if we will grow the sample The new code is clearer and will make the next update simpler. Differential Revision: https://phab.mercurial-scm.org/D10232

debugdiscovery: document relevant config option These config option are here for people to experiment with it. So lets document them in the command that is used for experimentation. Differential Revision: https://phab.mercurial-scm.org/D10231

re2: feed unicode string to re2 module when necessary My previous test were using the `pyre2` Python project, that wrap the Google RE2 library in python as a `re2` module and accept bytes as input. However the `fb-re2` Python project is also offering a wrapping of the Google RE2 library in python as a `re2` module ans accept only unicode on python3. So we detect this situation and convert thing to unicode when necessary. Hooray… We should consider using a rust wrapping for regexp handling. We needs regexps in Rust anyway and this give use more control with less variants and more sanity. Differential Revision: https://phab.mercurial-scm.org/D10284

rebase: skip obsolete commits even if they have pruned successors Issue 5782 reported that `hg rebase -r <obsolete commit with pruned successor>` failed with an error saying that it would cause divergence. Commit b7e2cf114e85 (rebase: do not consider extincts for divergence detection (issue5782), 2018-02-09) fixed it by letting you rebase the commit. However, that fix seems inconsistent with how we handle `hg rebase -r <pruned commit>`. To me, it should make no difference whether a commit is pruned itself or if it has (only) pruned successors. This patch changes it so we treat these two kinds of commits the same way. I let the message we print remain "note: not rebasing <commit>, it has no successor" even though that last part is not technically correct for commits with pruned successors. I doubt it will confuse users. Differential Revision: https://phab.mercurial-scm.org/D10240

tests: ask any chg instance to terminate before looking at sqlite dbs There are spurious errors in CI where the database is still locked, so force the daemon to quit to get deterministic behavior. Since the kill command itself is racy, also sleep 2s to give the server time to wake up and exit. Differential Revision: https://phab.mercurial-scm.org/D10244

chg: kill trailing comma in SEE ALSO Differential Revision: https://phab.mercurial-scm.org/D10243

bisect: use standard one-line commit summary This makes bisect use the standardized support for one-line commit summary I added a while back. That means that it will respect the `command-templates.oneline-summary` config. If also means that the default output now includes the first line of the commit message (see test impact). Differential Revision: https://phab.mercurial-scm.org/D10245

rhg: Add more conversions between error types This allows using the `?` operator in more places, as the next commit does. Differential Revision: https://phab.mercurial-scm.org/D10238