rust-dirstate-status: add `walk_explicit` implementation, use `Matcher` trait This is the first time we actually use the `Matcher` trait, still for a small subset of all matchers defined in Python. While I haven't yet actually measured the performance of this, I have tried to avoid any unnecessary allocations. This forces the use of heavy lifetimes annotations which I am not sure we can simplify, although I would be happy to be proven wrong. Differential Revision: https://phab.mercurial-scm.org/D7529

rust-matchers: add `FileMatcher` implementation Mercurial defines an `exactmatcher`, I find `FileMatcher` to be clearer, but am not opposed to using the old name. This change also switched the order of `assert_eq` arguments as it is clearer that way for most people. Differential Revision: https://phab.mercurial-scm.org/D7528

exchange: ensure all outgoing subrepo references are present before pushing We've run into occasional problems with people committing a repo, and then amending or rebasing in the subrepo. That makes it so that the revision in the parent can't be checked out, and the problem gets propagated on push. Mercurial already tries to defend against this sort of dangling reference by pushing *all* subrepo revisions first. This reuses the checks that trigger warnings in `hg verify` to bail on the push unless using `--force`. I thought about putting this on the server side, but at that point, all of the data has been transferred, only to bail out. Additionally, SCM Manager hosts subrepos in a location that isn't nested in the parent, so normal subrepo code would complain that the subrepo is missing when run on the server. Because the push command pushes subrepos before calling this exchange code, a subrepo will be pushed before the parent is verified. Not great, but no dangling references are exchanged, so it solves the problem. This code isn't in the loop that pushes the subrepos because: 1) the list of outgoing revisions is needed to limit the scope of the check 2) the loop only accesses the current revision, and therefore can miss subrepos that were dropped in previous commits 3) this code is called when pushing a subrepo, so the protection is recursive I'm not sure if there's a cheap check for the list of files in the outgoing bundle. If there is, that would provide a fast path to bypass this check for people not using subrepos (or if no subrepo changes were made). There's probably also room for verifying other references like tags. But since that doesn't break checkouts, it's much less of a problem. Differential Revision: https://phab.mercurial-scm.org/D7616

procutil: try and avoid angering CoreFoundation on macOS We've seen failures like this: objc[57662]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. objc[57662]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug. I think this is due to forking off some background processes during `hg update` or similar. I don't have any conclusive proof this is the fork() call that's to blame, but it's the most likely one since the regular `hg update` codepath uses the other fork() invocation (via workers) and we don't get this report from non-Google macOS users. Ugh. Differential Revision: https://phab.mercurial-scm.org/D7615

nodetree: simplify a conditionnal in shortesthexnodeidprefix instead of try to catch some attribute error, we could just nicely look if the attribute will be available. This make the code simpler to follow and less error prone since we no longer rely on a wider attribute catching. Differential Revision: https://phab.mercurial-scm.org/D7651

config: close file even if we fail to read it If we get an exception from cfg.read(), we would not close the file before this patch. This patch uses a context manager to make sure we close it. Differential Revision: https://phab.mercurial-scm.org/D7626

config: catch intended exception when failing to parse config When a new config parser was introduced in fca54469480e (ui: introduce new config parser, 2009-04-23), the reading side would raise a ConfigError which was then caught in the ui code. Then, in 2123aad24d56 (error: add new ParseError for various parsing errors, 2010-06-04), a ParseError was raised instead, but the call site was not updated. Let's start catching that ParseError. We still don't print it in a friendly way, but that's not worse than before. Differential Revision: https://phab.mercurial-scm.org/D7625

rust-hg-path: implement more readable custom Debug for HgPath{,Buf} The default prints the vector of bytes as a list of integers. I considered instead getting rid of the Debug trait, but we use the Debug format in lots of derived Debug instances, so we probably do want to implement it. Differential Revision: https://phab.mercurial-scm.org/D7604

util: implement sortdict.insert() As flagged by pytype (reported via Matt Harbison, thanks). This was broken by bd0fd3ff9916 (util: rewrite sortdict using Python 2.7's OrderedDict, 2017-05-16). We actually call insert() on namespaces.py:100, but we clearly don't have test coverage of that an no users have reported it AFAIK. Differential Revision: https://phab.mercurial-scm.org/D7680

patch: make __repr__() return str Caught by pytype: line 969, in __repr__: Function bytes.join was called with the wrong arguments [wrong-arg-types] Expected: (self, iterable: Iterable[bytes]) Actually passed: (self, iterable: Iterator[str]) Differential Revision: https://phab.mercurial-scm.org/D7682

pytype: suppress warnings about no 'open_binary' on importlib.resources Fixes these pytype warnings: line 43, in <module>: No attribute 'open_binary' on module 'importlib.resources' [module-attr] line 47, in open_resource: No attribute 'open_binary' on module 'importlib.resources' [module-attr] For some reason, I can't upgrade from 3.6.8 in my WSL environment. Differential Revision: https://phab.mercurial-scm.org/D7681

windows: if username(uid=None) is loaded, just use getpass This is at least consistent with what we do on other platforms in the base case. I don't know enough about Windows to fill in other cases that might exist here, but this at least should be a start. Differential Revision: https://phab.mercurial-scm.org/D7679

transplant: use check_incompatible_arguments() Differential Revision: https://phab.mercurial-scm.org/D7663

bookmarks: use check_incompatible_arguments() for inactive+action Differential Revision: https://phab.mercurial-scm.org/D7662

bookmarks: use cmdutil.check_incompatible_arguments() for action+rev Differential Revision: https://phab.mercurial-scm.org/D7648

bookmarks: use cmdutil.check_at_most_one_arg() for action Differential Revision: https://phab.mercurial-scm.org/D7647

rebase: use cmdutil.check_at_most_one_arg() for action Here we also needed to know what the action was (if any), so I've updated the helper to return any specified option. Differential Revision: https://phab.mercurial-scm.org/D7640

releasenotes: extract helper for checking for incompatible arguments This patch extracts a new check_incompatible_arguments() function similar to check_at_most_one_arg(). The difference is that the new function is for checking for arguments that are disallowed together with some other argument but not mutually exclusive among themselves. Differential Revision: https://phab.mercurial-scm.org/D7639

fix: use cmdutil.check_at_most_one_arg() Differential Revision: https://phab.mercurial-scm.org/D7638

patchbomb: use cmdutil.check_at_most_one_arg() Differential Revision: https://phab.mercurial-scm.org/D7637

export: use cmdutil.check_at_most_one_arg() Differential Revision: https://phab.mercurial-scm.org/D7636

amend: use cmdutil.check_at_most_one_arg() Differential Revision: https://phab.mercurial-scm.org/D7635

commit: use cmdutil.check_at_most_one_arg() Differential Revision: https://phab.mercurial-scm.org/D7634

clone: extract helper for checking mutually exclusive args We have some duplicated code for aborting if the user provided mutually exclusive arguments. Extensions surely have more such code. We also have duplicated translations and inconsistent output in this area. This patch introduces a simpler helper for checking if more than one option among a given set was given on the command line. I've made the clone code call the function to show that it works. The function has no good way of checking arguments with hyphens in them. I'll add that later if necessary. The function still won't be applicable in all cases, but I think it's still better than nothing. Differential Revision: https://phab.mercurial-scm.org/D7633

dirstate: when calling rebuild(), avoid some N^2 codepaths I had a user repo with 200k files in it. Calling `hg debugrebuilddirstate` took tens of minutes (I didn't wait for it). In that situation, changedfiles==allfiles, and both are lists. This meant that we had to run an average of 100k comparisons, for each of 200k files, just to check whether a file needed to have normallookup called (it always did), or drop. While it's probably not a huge issue, in my very awkward synthetic benchmark I wrote (not using a benchmark library or anything), I was seeing some slowdowns for small-changedfiles and very-large-allfiles invocations, with an inflection somewhere around 10 items in changedfiles (regardless of the size of allfiles); above 10 items in changedfiles, the new code appears to always be faster. For the case of 50k files in changedfiles and the same items in allfiles, I'm seeing differences of 15s of just running comparisons vs. 0.003793s. I haven't bothered to run a comparison of 200k items in changedfiles and allfiles. :) Differential Revision: https://phab.mercurial-scm.org/D7665

rust-warnings: fix warnings in tests It turns out that I also missed those warnings inside tests. This should be the last of them. One day we will get rid of this interface anyway. Differential Revision: https://phab.mercurial-scm.org/D7678

relnotes: mention the merging of index and nodemap

crecord: remove toggleamend Previous commit removed its only calling site.

crecord: repurpose "a" key to toggle all selections (BC) I really don't like "a". I keep accidentally hitting it when I actually want "A", and then I'm suddenly in a state I don't want to be in. There's a big wall of text telling me that I've turned amend mode on or off (which one was I orginally in?), and this seems very useless. If I wanted to amend or not, I would have chosen that from the command-line, not change my mind after I've already started picking hunks apart. Furthermore, for most uses of the hunk selector (revert, uncommit, shelve/unshelve), this amend toggle doesn't make sense. It seems much better to repurpose this key to be a "weaker" version of "A". It toggles all selections. This is pretty harmless if hit accidentally, (can just hit "a" again to toggle everything and undo it), and has immediate visual feedback that something happened: all the x's and blank spaces get switched around. And unlike with amend, the current flipped state is also immediately visible without having to read a wall of text. I'm calling this a BC, however, because somewhere, someone out there has probably really fallen in love with the old use of "a" and will get angry that we took it away.

hgweb: fix error in docstring Despite the subtle semantic difference, this sentence really meant to say "overridden", not "overwritten".

crecord: rewrite help string to avoid mentioning "crecord" Despite its heritage, "crecord" is now mostly Mercurial-internal jargon. I find it better to call it "the curses hunk selector". Also slightly rewrote the part about which commands can use it. While I do believe that commit, shelve, and revert are the only commands in core that can use it, Evolve also adds at least amend and uncommit to the list.

update: add some tests for the status quo of morestatus on update conflicts Differential Revision: https://phab.mercurial-scm.org/D7666

typing: suppress a warning that mercurial.windows.checkosfilename is missing This function is used to check filename portability everwhere, so it isn't tucked into the windows.py module. I supposed the alternative is to move it and then alias it in `util`. I'm guessing it was done like this to save an import. Differential Revision: https://phab.mercurial-scm.org/D7677

typing: add an assertion to util.versiontuple Fixes the following warning: line 1177, in versiontuple: No attribute 'split' on None [attribute-error] In Optional[bytes] Differential Revision: https://phab.mercurial-scm.org/D7676

typing: add a couple of assertions to lrucachedict to help pytype Fixes the following warnings: line 1488, in pop: No attribute 'markempty' on None [attribute-error] In Optional[Union[Any, _lrucachenode, nothing]] line 1578, in popoldest: No attribute 'markempty' on None [attribute-error] In Optional[Union[Any, _lrucachenode, nothing]] Differential Revision: https://phab.mercurial-scm.org/D7674

util: rename a variable to avoid confusing pytype Fixes the following warning: line 1205, in f: No attribute 'append' on Dict[nothing, nothing] [attribute-error] In Union[Dict[nothing, nothing], list] Differential Revision: https://phab.mercurial-scm.org/D7673

pytype: suppress the import-error in util.py when importing re2 Differential Revision: https://phab.mercurial-scm.org/D7672

debugcommands: finish moving `extendeddateformats` from util to dateutil Caught by pytype. Differential Revision: https://phab.mercurial-scm.org/D7671

ui: convert exception data to bytes when printing chained exception info Caught by pytype. Differential Revision: https://phab.mercurial-scm.org/D7670

ui: use a context manager to handle file streams in edit() Differential Revision: https://phab.mercurial-scm.org/D7669

graft: reuse cmdutl.resolvecommitoptions() Differential Revision: https://phab.mercurial-scm.org/D7632

rebase: make rebase not crash if p1 == p2 != nullid While this shouldn't happen normally, some historical bugs can have caused this kind of commit to exist. Instead of crashing and having it be unobvious how to recover, let's try to continue on. Without this, we get an error like "ValueError: min() arg is an empty sequence" Differential Revision: https://phab.mercurial-scm.org/D7664

merge: add commands.merge.require-rev to require an argument to hg merge This is related to commands.rebase.requiredest, commands.update.requiredest, and commands.push.require-revs. Since it isn't really a "destination", I went with require-rev to be similar to push's require-revs. Differential Revision: https://phab.mercurial-scm.org/D7620

tests: make test-config-env.py a little less hacky Overriding os.path.isdir to always returns False is very confusing. Let's make it more explicit by specifically not returning any defaultrc/ configs. Differential Revision: https://phab.mercurial-scm.org/D7623

help: get helptext/ data from `resources` module if available For PyOxidizer, we need to read configs using the `resources` module. This patch makes it so we use that module if available (i.e. Python >= 3.7). It does that by adding a new `open_resource()` function to our `resourceutil` module. Tested by running `$PYTHON ./hg help pager` for each $PYTHON in {python2, python3.6, python3.7}. Differential Revision: https://phab.mercurial-scm.org/D7622

pycompat: allow pycompat.sysbytes() even if input already is bytes pycompat.sysstr() on py3 accepts an input that's already str (i.e. unicode). This patch makes it so pycompat.sysbytes() on py3 accepts an input that's already bytes. Allowing that makes it possible to do pycompat.sysbytes(fp.name) where fp.name is either bytes or unicode, as we'll get when fp can come from either open() or resources.open_binary(). Differential Revision: https://phab.mercurial-scm.org/D7621

merge with stable

rust-cpython: do not convert warning pattern to utf-8 bytes On Unix, both Rust Path and Mercurial expect a locale-dependent bytes, and we don't support Windows yet. Differential Revision: https://phab.mercurial-scm.org/D7614

rust-cpython: import utils::files::* function at module level IIRC, it's common in Rust to call functions with the module prefix. Differential Revision: https://phab.mercurial-scm.org/D7613

py3: send bytes from Rust-created warning patterns Python code expects bytes in both Python 2 and Python 3, so we should send bytes. Differential Revision: https://phab.mercurial-scm.org/D7612

py3: pass bytes to `configint` and `configbool` Both functions require bytes, even in Python 3. Differential Revision: https://phab.mercurial-scm.org/D7611

rust-threads: force Rayon to respect the worker count in config As per the inline comment, this is a workaround because Rust code does not yet know how to read config files. Differential Revision: https://phab.mercurial-scm.org/D7610

rust-dirs: handle forgotten `Result`s In 1fe2e574616e I introduced a temporary bugfix to align Rust code with a new behavior from C/Python and forgot about a few `Result`s (cargo's compiler cache does not re-emit warnings on cached modules). This fixes it. For the record, I am still unsure that this behavior change is a good idea. Note: I was already quite unhappy with the setters and getters for the `DirstateMap` and, indirectly, `Dirs`, and this only further reinforces my feelings. I hope we can one day fix that situation at the type level; Georges Racinet and I were just talking about devising a POC for using the builder pattern in the context of FFI with Python, we'll see what comes out of it. Differential Revision: https://phab.mercurial-scm.org/D7609

merge with stable

status: outputting structured unfinished-operation information This adds a new item in the json/template output for morestatus and added item types to all entries. Differential Revision: https://phab.mercurial-scm.org/D7595

chg: fix chg to work with py3.7+ "coercing" the locale When the environment is empty (specifically: it doesn't contain LC_ALL, LC_CTYPE, or LANG), Python will "coerce" the locale environment variables to be a UTF-8 capable one. It sets LC_CTYPE in the environment, and this breaks chg, since chg operates by: - start hg, using whatever environment the user has when chg starts - hg stores a hash of this "original" environment, but python has already set LC_CTYPE even though the user doesn't have it in their environment - chg calls setenv over the commandserver. This clears the environment inside of hg and sets it to be exactly what the environment in chg is (without LC_CTYPE). - chg calls validate to ensure that the environment hg is using (after the setenv call) is the one that the chg process has - if not, it is assumed the user changed their environment and we should use a different server. This will *never* be true in this situation because LC_CTYPE was removed. Differential Revision: https://phab.mercurial-scm.org/D7550

fuzz: add support for fuzzing under either Python 2 or 3 This was more of a hairball than I hoped, but it appears to work. The hg-py3 branch of my oss-fuzz fork on github has the remaining changes to switch us to Python 3, but we may as well retain Python 2 fuzzing support for at least a little while. Differential Revision: https://phab.mercurial-scm.org/D7592

phabricator: color the status in the "phabstatus" view I couldn't figure out strikethrough for "abandoned" like I've see with word diff. Differential Revision: https://phab.mercurial-scm.org/D7608

phabricator: add the "Changes Planned" status name I noticed this when coloring the status output. But it looks like status names are also used in the pseudo revset language as well, and abort when parsing a name not in this list. Differential Revision: https://phab.mercurial-scm.org/D7607

fuzz: fix mpatch_corpus to not have an overridden __repr__ on py3 Differential Revision: https://phab.mercurial-scm.org/D7606