context: avoid writing outdated dirstate out (issue5584) Before this patch, workingctx.status() may cause writing outdated dirstate out, if: - .hg/dirstate is changed simultaneously after last loading it, - there is any file, which should be dirstate.normal()-ed Typical issue case is: - the working directory is updated by "hg update" - .hg/dirstate is updated in background (e.g. fsmonitor) This patch compares identities of dirstate before and after acquisition of wlock, and avoids writing outdated dirstate out, if change of .hg/dirstate is detected.

tests: factor external procedures out for portability Fortunately, "&&" is treated as "execute next, if previous doesn't fail" both on POSIX and Windows. But keeping portability of "dirstaterace.command" manually is troublesome. This patch factors external procedures out as a shell script for portability. "sh SCRIPT" always allows scripting in POSIX style. This change is also for convenience. Fixed script name can reduce command line arguments. "r" prefix is needed for "sh '$TESTTMP/dirstaterace.sh'", because $TESTTMP contains backslash on Windows.

dirstate: add identity information to detect simultaneous changing in storage This identity is used to examine whether dirstate is simultaneously changed in storage after previous caching (see issue5584 for detail). util.cachestat can't be used for this purpose, because it has no valuable information on Windows. On the other hand, util.filestat can detect changing dirstate in storage certainly, regardless of platforms. https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan Strictly speaking, if underlying filesystem doesn't support ctime/mtime, util.filestat can't detect simultaneous changing in storage as expected. But simultaneous changing on such (very rare) platform can't be detected regardless of this patch series. Therefore, util.filestat should be reasonable identity for almost all usecases.

util: make filestat.__eq__ return True if both of self and old have None stat For convenience to compare two filestat objects regardless of None-ness of stat field.

vfs: create copy at renaming to avoid file stat ambiguity if needed In order to fix issue5418, bff5ccbe5ead made vfs.rename(checkambig=True) omit advancing mtime of renamed file, if renamed file is owned by another (EPERM is raised in this case). But this omission causes rewinding mtime at restoration in such situation, and makes avoiding file stat ambiguity difficult, because ExactCacheValidationPlan assumes that mtime should be advanced, if a file is changed in same ctime. https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan Ambiguity of file stat also requires issue5584 to be fixed with other than file stat, but "hash of file", "generation ID" and so on were already rejected ideas (please see original RFC linked from "Outline of issue" in ExactCacheValidationPlan page). This omission occurs: - only for non append-only files (dirstate, bookmarks, and phaseroots), and - only if previous transaction is rollbacked by another user The latter means "sharing a repository clone via group permission". This is reasonable usecase, but not ordinary for many users, IMHO. "hg rollback" itself has been deprecated since Mercurial 2.7, too. Therefore, increasing the cost at rollbacking previous transaction executed by another a little seems reasonable, for avoidance of file stat ambiguity. This patch does: - create copy of (already renamed) source file, if advancing mtime fails for EPERM - rename from copied file to destination file, and - advance mtime of renamed file, which is now owned by current user This patch also factors "self.join(src)" out to reduce redundancy.

vfs: factor out "rename and avoid ambiguity" to reuse This makes subsequent patch simple.

util: make filestat.avoidambig() return whether ambiguity is avoided or not

debugcommands: issue warning when repo has secret changesets (issue5589) This seems like a prudent thing to do. As the inline comment says, we may want to make this abort once the functionality is stabilized as part of `hg bundle`. Let's save that debate for another day.

streamclone: consider secret changesets (BC) (issue5589) Previously, a repo containing secret changesets would be served via stream clone, transferring those secret changesets. While secret changesets aren't meant to imply strong security (if you really want to keep them secret, others shouldn't have read access to the repo), we should at least make an effort to protect secret changesets when possible. After this commit, we no longer serve stream clones for repos containing secret changesets by default. This is backwards incompatible behavior. In case anyone is relying on the behavior, we provide a config option to opt into the old behavior. Note that this defense is only beneficial for remote repos accessed via the wire protocol: if a client has access to the files backing a repo, they can get to the raw data and see secret revisions.

json: pass formatting options recursively This bug was introduced in 654e9a1c8a6c. It's okay to escape <>, but is unnecessary for command output.

json: avoid extra string manipulation of dict keys A key must be string per JSON spec, and that's also true for template dicts.

test-obsolete: include <> in user field to check JSON escapes I found json() filter doesn't pass formatting options recursively. That's why <> are escaped.

templatefilers: correct filename in header comment

repoview: remove special casing of "requirements" At the time this code was introduced (3a6ddacb7198), the inline comment was true. This changed in e3a928bd1cd4. The proxy is no longer needed.

bookmarks: move variable initialization earlier Since we no longer set '_clean = False' during the initialization loop, we can move the attribute assignment earlier in the function for clarity. (no speed improvement expected or measured ;-) )

bookmarks: directly use base dict 'setitem' The bmstore '__setitem__' method is setting an extra flag that is not needed during initialization. Skipping the method will allow further cleanup and yield some speedup as a side effect. Before: ! wall 0.009120 comb 0.010000 user 0.010000 sys 0.000000 (best of 312) After: ! wall 0.007874 comb 0.010000 user 0.010000 sys 0.000000 (best of 360)

bookmarks: rely on exception for malformed lines Since we already have an exception context open, for other thing, we can simplify the code a bit and rely on exception handling for invalid lines. Speed is not the main motivation for this changes. However as I'm in the middle of benchmarking things we can see a small positive impact. Before: ! wall 0.009358 comb 0.000000 user 0.000000 sys 0.000000 (best of 303) After: ! wall 0.009173 comb 0.010000 user 0.010000 sys 0.000000 (best of 310)

bookmarks: explicitly convert to 'node' during initialization We know the content of the file is supposed to be full hex. So we can do the translation ourselves and directly check if the node is known. As nice side effect we now have proper error handling for invalid node value. Before: ! wall 0.021580 comb 0.020000 user 0.020000 sys 0.000000 (best of 134) After: ! wall 0.009342 comb 0.010000 user 0.010000 sys 0.000000 (best of 302)

bookmarks: prefetch 'lookup' outside of the loop Skipping the attribute lookup up raise a significant speedup. Example on a repository with about 4000 bookmarks. Before: ! wall 0.026027 comb 0.020000 user 0.020000 sys 0.000000 (best of 112) After: ! wall 0.021580 comb 0.020000 user 0.020000 sys 0.000000 (best of 134) (This is also in its own changeset to clarify the perf win from another coming changesets)

perf: add a perfbookmarks command A new command dedicated to benchmark of bookmark initialization.

perfphases: add a flag to also include file access time The flag purges all phases data so we'll have to read the file from disk again.

perf: have a generic "clearstorecache" function There are multiple places where we will want to purge some store cache. So we promote the existing _clearobsstore function to a something reusable.

localrepo: move filtername to __init__ This is obviously an instance attribute, not a type attribute. The modern Python style is to use __init__ for defining these. This exposes statichttprepo as inheriting from localrepository without calling its __init__. As a result, its __init__ defines a lot of variables that methods on localrepository's methods need. But factoring the common bits into a separate class is for another day.

obsolete: move obsstore creation logic from localrepo This code has more to do with obsolete.py than localrepo.py. Let's move it there.

match: allow pats to be None match.match already interprets "!bool(patterns)" as matching everything (but includes and excludes still apply). We might as well allow None, which lets us simplify some callers a bit. I originally wrote this patch while trying to change match.match(patterns=[]) to mean to match no patterns. This patch is one step towards that goal. I'm not sure it'll be worth the effort to go all the way there, but I think this patch still makes sense on its own.

tests: add a test for installing hg with pip in a virtualenv Since we're doing so much clever junk in our setup.py, let's have a test that exercises it. Thanks to Matt Harbison for testing this on Windows and verifying that installenv/*/hg would work as a way to work around bin being called Scripts on Windows.

hghave: add check for virtualenv

setup: introduce dummy copies of setuptools flags Since we're filtering out some egg gunk, we need to emulate these flags which disable eggs so that pip still works.

extensions: catch uisetup and extsetup failures and don't let them break hg Otherwise users of the patience diff extension will be unable to run anything at all in hg 4.3 until they figure out what's broken.

tests: add test demonstrating how broken third-party extensions can get I intend to fix this, but will do the fix as a separate change to make the behavior change obvious. This was inspired by some users having the patience diff extension, which broke when we moved bdiff.so so thoroughly the users can't even run 'hg debuginstall'.