merge: check created file dirs for path conflicts only once (issue5716) In large repositories, updates involving the creation of many files check the same directories repeatedly in the wctx manifest. Move these checks out to a separate loop to avoid repeated checks hitting the manifest. Differential Revision: https://phab.mercurial-scm.org/D1226

merge: cache unknown dir checks (issue5716) As mentioned in D1222, the recent pathconflicts change regresses update performance in large repositories when many files are being updated. To mitigate this, we introduce two caches of directories that have already found to be either: - unknown directories, but which are not aliased by files and so don't need to be checked if they are files again; and - missing directores, which cannot cause path conflicts, and cannot contain a file that causes a path conflict. When checking the paths of a file, testing against this caches means we can skip tests that involve touching the filesystem. Differential Revision: https://phab.mercurial-scm.org/D1224

dispatch: add HGPLAIN=+strictflags to restrict early parsing of global options If this feature is enabled, early options are parsed using the global options table. As the parser stops processing options when non/unknown option is encountered, it won't mistakenly take an option value as a new early option. Still "--" can be injected to terminate the parsing (e.g. "hg -R -- log"), I think it's unlikely to lead to an RCE. To minimize a risk of this change, new fancyopts.earlygetopt() path is enabled only when +strictflags is set. Also the strict parser doesn't support '--repo', a short for '--repository' yet. This limitation will be removed later. As this feature is backward incompatible, I decided to add a new opt-in mechanism to HGPLAIN. I'm not pretty sure if this is the right choice, but I'm thinking of adding +feature/-feature syntax to HGPLAIN. Alternatively, we could add a new environment variable. Any bikeshedding is welcome. Note that HGPLAIN=+strictflags doesn't work correctly in chg session since command arguments are pre-processed in C. This wouldn't be easily fixed.

fancyopts: add early-options parser compatible with getopt() The next patch will add a flag for strict parsing of early options, where we'll have to parse all early options at once instead of processing them one-by-one by dispatch._earlygetopt(). That's why I decided to hook fancyopts(). All dispatch._early*opt() functions is planned to be replaced with this function. But in this stable series, only the strict mode will be handled by fancyopts.earlygetopt().

largefiles: pay attention to dropped standin files when updating largefiles Previously, the largefile for a dropped standin would be deleted here, and then restored from the cache. This had the effect of clobbering uncommitted changes if a revert caused the file to be forgotten, which is not what happens with a normal file. Now the removal and update is skipped for dropped largefiles, and the corresponding standin is deleted from disk. This was noticed when working on issue5738 because the forgotten standin files were left behind, and that changes the behavior of the next rename to that directory. My first attempt was to cleanup the standins before calling this. That failed, because this function deletes the largefile if the corresponding standin is missing. This function is called by the revert command, merge (and therefore update), and patch, via the scmutil.marktouched() override. So it should be pretty narrow in scope. I didn't mark issue5738 as fixed because the move related issues can still happen if the main tree and the .hglf subtree get out of sync somehow. I don't see an easy fix for that, but that should be an edge case. If whoever queues this thinks it is good enough to close out the bug and can cram it into the summary, go for it.

test-largefiles: demonstrate problems with renaming and reverting a directory These things were uncovered looking at issue5738. First, if the destination directory exists under .hglf, the source is moved under the destination instead of renaming the last component for `hg mv srcdir dstdir`. This is extra confusing, because it occurs even if the user visible destination (i.e. the path _not_ under .hglf) does not exist. Additionally, when a largefile is forgotten via revert, any modifications end up getting clobbered. For normal files, the forgotten file is left unchanged, as shown by test-import.t. The forget command on a largefile will correctly leave the file unmodified.

tests: add globs for Windows

cat: record the current behavior of wildcard matches in subrepos Mercurial subrepos support any match patterns.

match: remove doc about undefined behavior of visitdir() This was added by 8545bd381504, but core matchers support visitdir() of arbitrary locations since 2773540c3650, and verifier._verifymanifest() doesn't seem to strictly obey the restriction. I have no idea how important this API contract is for third-party extensions. That's why this patch is RFC.

merge with stable