Merge with crew

osutil: factor out creation and init of listdir_stat

dirstate: avoid use of zip on big lists In a clean working directory containing 170,000 tracked files, this improves performance of "hg --time diff" from 1.69 seconds to 1.43. This idea is due to Siddharth Agarwal.

dirstate: move file type filtering to its source This prepares us to move to a much faster statfiles implementation on Unix.

clfilter: rename `unfilteredmeth` to `unfilteredmethod` As originally intended.

clfilter: fix a false positive in the test-obsolete.t We push between two repo which once filtered looks unrelated. Weakness in the current implementation allows this push to be done without -f. But later improvement with filtering will make this push fails for unrelatedness. However we want this push to fail for including bumped changeset. So we had a smaller push --force to make them related.

clfilter: ensure that filecache on localrepo is unfiltered All filecache usage on repo is for logic that should be unfiltered. The caches should be common to all filtered instances, and computation must be done unfiltered. A dedicated storecache subclass is created for this purpose.

clfilter: add a propertycache that must be unfiltered Some of the localrepo property caches must be computed unfiltered and stored globally. Some others must see the filtered version and store data relative to the current filtering. This changeset introduces two classes `unfilteredpropertycache` and `filteredpropertycache` for this purpose. A new function `hasunfilteredcache` is introduced for unambiguous checking for cached values on unfiltered repos. A few tweaks are made to the property cache class to allow overriding the way the computed value is stored on the object. Some logic relative to _tagcaches is cleaned up in the process.

largefile: status is buggy on repoproxy, so run unfiltered For some yet-unkown reason, largefile status does not work on repoproxy. As status is not affected by filtering, we run it unfiltered. Na'Tosha Bard's view on this issue: "but, well, largefiles status is kind of an unholy piece of code"

clfilter: mq should not warn about filtered mq patches MQ warns when qstatus contains unknown nodes. With changelog filtering, a node may be unknown because it is filtered. Thus, an unfiltered repo is used for this check.

clfilter: ensure that mq performs commits on unfiltered repos

clfilter: use unfiltered repo for bookmark push logic The remote location of the bookmark may be filtered locally. This changeset ensures that bookmark movement logic has access to all the repo's content.

clfilter: `bookmark.validdest` should run on unfiltered repo The logic recently added to `bookmark.validdest` uses data about obsolete changesets to see if a bookmark destination is valid. Obsolete changesets are likely to be filtered, so we need to work on an unfiltered repository.

clfilter: unfilter some parts of the push logic Computation of common changesets during push needs to be done on the widest set possible. An unfiltered version of the repo is kept for discovery and various revset calls. The discovery code itself enforces the filtering of unserved outgoing changeset.

subrepo: add argument to "diff()" to pass "ui" of caller side (issue3712) (API) Color extension achieves colorization by overriding the class of "ui" object just before command execution. Before this patch, "diff()" of abstractsubrepo and classes derived from it has no "ui" argument, so "diff()" of hgsubrepo uses "self._repo.ui" to invoke "cmdutil.diffordiffstat()". For separation of configuration between repositories, revision 573bec4ab7ba changed the initialization source of "self._repo.ui" from "ui"(overridden) to "baseui"(plain) of parent repository. And this caused break of colorization. This patch adds "ui" argument to "diff()" of abstractsubrepo and classes derived from it to pass "ui" object of caller side.

clfilter: prevent unwanted warning about filtered parents as unknown During changectx __init__ the dirstate's parents MAY be checked. If the repo is filtered, this check will complain "working directory has unknown parents" even if the parents are perfectly known. This may happen when the repo is used for serving and the dirstate has parents that are secret, as those secret changesets will be filtered.

clfilter: strip logic should be unfiltered Strip is a "write" operation that needs to be aware of the whole repo's content before destroying changesets. Only the low level function is altered. The top level command will still process its argument filtered (if any filtering is in place).

clfilter: verify logic should be unfiltered To verify a changelog obviously needs all of it. The verify logic now ensures it works on an unfiltered repository.

clfilter: phases logic should be unfiltered Phase computations and boundary movements need to be aware of all revisions that exist in the repository to return correct results.

clfilter: unfilter computation of obsolescence related computation All obsolescence related sets need to be computed on the full unfiltered version of the repository, in particular because several of them (obsolete, extinct) are used to compute the hidden revisions. On a filtered repo, revset predicates related to these sets will be properly filtered because of revset's own pre-filtering.

clfilter: ensure changeset creation in the repo is run unfiltered This applies to both creation of new commits and application of changegroups.

clfilter: ensure changegroup generation is run unfiltered Changegroup generation logic needs to be aware of the whole repository to work properly.

clfilter: ensure `rollback` is run unfiltered Rollback logic needs to be aware of the whole repository to work properly.

clfilter: ensure cache invalidation is done on the main unfiltered repo The proxy version will not hold any cache for now. But we have to ensure all cache operations are done on the unfiltered version.

clfilter: ensure that tag logic runs unfiltered The current tag logic is not aware of filtering. We keep the status quo, ensuring that the tag cache is computed as before: without any filtering.

clfilter: ensure `branchcache` logic runs unfiltered The current branchcache construction is not aware of filtering. We keep the status quo, ensuring that the branch cache logic is computed as before: without any filtering.

clfilter: introduce an `unfilteredmethod` decorator This decorator ensure the method in run on an unfiltered version of the repository. See follow-up commit for details. This decorator is not named `unfiltered` because it would clash with the `unfilteredmethod` on `localrepo` itself.

clfilter: introduce an "unfiltered" method on localrepo This commit is part of the changelog level filtering effort. It returns the main "unfiltered" version of a repo-like object. For localrepo this means the same localrepo object. But this method will be overwritten by the filtered versions of a repository to return the core unfiltered version of the repo. Introducing this simple method first allows later commits to prepare for the use of a filtered version of a repository. A new repo method is added because a lot of users may call it. At the end of this series of commits, about 40 calls exist in core and hgext.

clfilter: remove usage of `range` and `xrange` in scmutil.revrange For changelog level filtering to take effect it need to be used for any iteration. This changeset removes usage of `range` and `xrange` that survived the first pass.

hgweb: display diff for a changeset against any parents (issue2810) During merge of branches, it is useful to compare merge results against the two parents. This change adds this support to hgweb. To specify which parent to compare to, use rev/12300:12345 where 12300 is a parent changeset number. Two links are added to changeset web page so that one can choose which parent to compare to.

branch: add missing repo argument to checknewlabel scmutil.checknewlabel takes a repo object as its first argument. When the call to this function was added in e689b0d91546, the first argument was mistakenly set to 'None'.

Merge with stable.

rebase: fix pull --rev options clashing with --rebase (issue3619) Rebase also have a plain `--rev` option used to select the rebase set (as `--base` or `--source` would). But the content of the --rev option was intended for the remote repo and is irrelevant for the local rebase operation. We expect `hg pull --rebase` to stick with the default behavior here: hg rebase --base . --dest tip(branch(.)) The `rev` option is dropped from the option passed to rebase.

dirstate: remove obsolete comment from setbranch This comment should have been removed in e689b0d91546, when the call to scmutil.checknewlabel was removed.

merge with stable

tests: fix broken fix of test-phases output

update: allow update to existing branches with invalid names (issue3710) Starting with 361ab1e2086f, users are no longer able to update a working copy to a branch named with a "bad" character (such as ':'). Prior to v2.4, it was possible to create branch names using "bad" characters, so this breaks backwards compatibility. Mercurial must allow users to update to existing branches with bad names. However, it should continue to prevent the creation of new branches with bad names. A test was added to confirm that 'hg update' works as expected. The test uses a bundled repo that was created with an earlier version of Mercurial.

commit: increase perf by building a new addlist instead of editing the old one When commiting to a repo with lots of files (>170000), manifest.py:addlistdelta takes some time because it's editing a large array many times. Changing it to build a new array instead of editing the old one saves around 0.04 seconds on a 1.64 second commit. A 2.5% gain. The gain here is pretty minor, but it was blatantly at the top of the profiler report and the fix is straight forward. I tested it by comparing the arrays produced by the new and old logic while running all of the tests.

template engine: convert generator-based iterator to list-based iterator If a template iterator is implemented with generator, the iterator is exhau= sted after we use it. This leads to undesired behavior in template. This chang= e converts a generator-based iterator to list-based iterator when template en= gine first detects a generator-based iterator. All future usages of iterator wi= ll use list instead.

command: remove phase from the list of basic command This is not a basic command. There is no reason new user should needs to know about it. Thanks to Matt Mackall for pointing this.

merge with stable

phases: fix missing "error" module import (issue3707)

test-mq-qrefresh: test that the patch at qtip is indeed empty ...after qrefreshing a non-existent file

tests: only call check-code once The accepted warnings list is now empty, let's try to keep it that way.

ancestor: fix a comment (followup to 0b03454abae7)

revlog: allow reverse iteration with revlog.revs We often need to perform rev iteration in reverse order. This changeset makes it possible to do so, in order to avoid costly reverse or reversed() calls later.

convert: add config option to use the local time zone The default for the time zone offset in a converted changeset has always been 0 (UTC). With this patch, the converted changeset is modified so that the local offset from UTC is specified as the time zone offset. The option is specified as the boolean convert.localtimezone (default False). Example usage: hg convert -s cvs --config convert.localtimezone=True example-cvs example-hg IMPORTANT: the patch only applies to conversions from cvs or svn. The documentation for the option only appears in those two sections in the convert help text.

rebase: use revlog.findmissingrevs to compute detach set

revlog: add rev-specific variant of findmissing This will be used by rebase in an upcoming commit.

revlog: switch findmissing to use ancestor.missingancestors This also speeds up other commands that use findmissing, like incoming and merge --preview. With a large linear repository (>400000 commits) and with one incoming changeset, incoming is sped up from around 4-4.5 seconds to under 3.

ancestor: faster algorithm for difference of ancestor sets One of the major reasons rebase is slow in large repositories is the computation of the detach set: the set of ancestors of the changesets to rebase not in the destination parent. This is currently done via a revset that does two walks all the way to the root of the DAG. Instead of doing that, to find ancestors of a set <revs> not in another set <common> we walk up the tree in reverse revision number order, maintaining sets of nodes visited from <revs>, <common> or both. For the common case where the sets are close both topologically and in revision number (relative to repository size), this has been found to speed up rebase by around 15-20%. When the nodes are farther apart and the DAG is highly branching, it is harder to say which would win. Here's how long computing the detach set takes in a linear repository with over 400000 changesets, rebasing near tip: Rebasing across 4 changesets Revset method: 2.2s New algorithm: 0.00015s Rebasing across 250 changesets Revset method: 2.2s New algorithm: 0.00069s Rebasing across 10000 changesets Revset method: 2.4s New algorithm: 0.019s

bisect: add example for limiting bisection to specified directories The bisect command does not have an option to limit itself only to subdirectories, but it's possible to use revsets for the --skip option for the same effect. Given the relative obscurity of revsets, it helps to have this as another example for bisect.

subrepo: use posixpath when diffing, for consistent paths This fixes a Windows failure in test-subrepo-recursion.t introduced by c84ef0047a94.

run-tests: fix an unnoticed check-code violation

run-tests: add a --compiler option Without this option, it is not possible to run the test suite on Windows using mingw's gcc as the compiler.

run-tests: make build command line less intimidating Use a dict for parameters to the format string, instead of a ridiculous number of positional parameters.

hooks: be even more forgiven of non-fd descriptors (issue3711) Looks like there are instances where sys.stdout/stderr contain file handles that are invalid. We should be tolerant of this for hook I/O redirection, as our primary concern is not garbling our own output stream.

hooks: delay I/O redirection until we actually run a hook (issue3711) We were attempting to redirect I/O even if no hook was actually getting called. This defers redirection until we've found something to do.

util: make chunkbuffer non-quadratic on Windows The old str-based += collector performed very nicely on Linux, but turns out to be quadratically expensive on Windows, causing chunkbuffer to dominate in profiles. This list-based version has been measured to significantly improve performance with large chunks on Windows, with negligible overall overhead on Linux (though microbenchmarks show it to be about 50% slower). This may increase memory overhead where += didn't behave quadratically. If we want to gather up 1G of data to join, we temporarily have 1G in our list and 1G in our string.

revset: backed out changeset 54cedee86e51 This was causing clones of the hg repo to go from 12.4s to 14.7s.