bookmark: use 'applychanges' when updating bookmark in histedit

bookmark: use 'applychanges' when updating a bookmark through pushkey

bookmark: use 'applychanges' when updating from a remote

bookmark: use 'applychanges' for adding new bookmark

bookmark: use 'applychanges' for bookmark renaming

bookmark: use 'applychanges' for bookmark deletion

bookmark: introduce a 'applychanges' function to gather bookmark movement We want to track bookmark movement within a transaction. For this we need a more centralized way to update bookmarks. For this purpose we introduce a new 'applychanges' method that apply a list of changes encoded as '(name, node)'. We'll cover all bookmark updating code to this new method in later changesets and add bookmark move in the transaction when all will be migrated.

obsstore: keep self._data updated with _addmarkers This makes sure obsstore._data is still correct with added markers. The '_data' propertycache was added in 17ce57b7873f.

match: make base matcher return True for visitdir If a matcher doesn't implement visitdir, we should be returning True so that tree traversals are not prematurely pruned. The old value of False would prevent tree traversals when using any matcher that didn't implement visitdir. Differential Revision: https://phab.mercurial-scm.org/D83

tests: fix an incorrect description in test-ignore.t Differential Revision: https://phab.mercurial-scm.org/D82

templatekw: hide {peerpaths} keyword for 4.3 Thinking a bit further about list/dict subscript operation (proposed by issue 5534), I noticed the current data structure, a dict of dicts, might not be ideal. For example, if there were "'[' index ']'" and "'.' key" operators, "{parents[0]}" would return "{p1rev}:{p1node}", and we would probably want to write "{parents[0].desc}" to get the first element of "{parents % "{desc}"}". This will basically execute parents[0].makemap()['desc'] in Python. Given the rule above, "{peerpaths.default.pushurl}" will be translated to peerpaths['default'].makemap()['pushurl'], which means {peerpaths} should be a single-level dict and sub-options should be makemap()-ed. "{peerpaths % "{name} = {url}, {pushurl}, ..."}" (Well, it could be peerpaths['default']['pushurl'], but in which case, peerpaths['default'] should be a plain dict, not a hybrid object.) So, let's mark the current implementation experimental and revisit it later.

parsers: fix invariant bug in find_deepest (issue5623) find_deepest is used to find the "best" ancestors given a list. In the main loop it keeps an invariant called 'ninteresting' which is supposed to contain the number of non-zero entries in the 'interesting' array. This invariant is incorrectly maintained, however, which leads the the algorithm returning an empty result for certain graphs. This has been fixed. Also, the 'interesting' array is supposed to fit 2^ancestors values, but is incorrectly allocated to twice that size. This has been fixed as well. The tests in test-ancestor.py compare the Python and C versions of the code, and report the error correctly, since the Python version works correct. Even so, I have added an additional test against the expected result, in the event that both algorithms have an identical error in the future. This fixes issue5623.

configitems: register the 'worker.backgroundclose' config

configitems: register the 'progress.width' config

configitems: register the 'color.pagermode' config

configitems: handle case were the default value is not static In some case, the default of one value is derived from other value. We add a way to register them anyway and an associated devel-warning. The registration is very naive for the moment. We might be able to have a better way for registering each of these cases but it could be done later.

bugzilla: move the default regexp for fix in the config declaration This mimic the change requested by Yuya for '_default_bug_re'.

configitems: register the 'bugzilla.version' config

configitems: register the 'bugzilla.usermap' config

configitems: register the 'bugzilla.user' config

configitems: register the 'bugzilla.timeout' config

configitems: register the 'bugzilla.template' config

configitems: register the 'bugzilla.style' config

configitems: register the 'bugzilla.strip' config

configitems: register the 'bugzilla.regexp' config The default value is moved from the class to the config registration.

changegroup: stop returning and recording added nodes in 'cg.apply' cg.apply used to returns the added nodes. Callers doesn't have a use for it anymore, remove the added node and stops recording it in the current operation. This information was added in the current release cycle so no extensions breakage should happens.

phases: remove trace of addednodes in the 'phase-heads' handling updatephases have no use of the 'addednodes' parameter since 50243c975fc2. However caller are still passing it for nothing, remove the parameter and remove computing of the added nodes in caller.

phases: test phases tracking at the transaction level Now that we have all tracking in place, the data in `tr.changes['phases']` dictionary should be correct and we should test it. It is a bit late in the cycle to discuss to add any public API (eg: hooks) that expose the data to the user, so we just add a small test extension displaying the data. It is enabled for the phases tests. New output have been manually checked for consistency.

phases: track phase changes from 'retractboundary' We adds new computation to find and record the revision affected by the boundary retraction. This add more complication to the function but this seems fine since it is only used in a couple of rare and explicit cases (`hg phase --force` and `hg qimport`). Having strong tracking of phase changes is worth the effort.

phases: detect when boundaries has been actually retracted It is useful to detect noop and avoid expensive operations in this case. We return the information to inform the caller of a possible update. Top level function might need to react to the phase update (eg: invalidating some caches, tracking phase change).

phases: rework phase movement code in 'cg.apply' to use 'registernew' We rework the code to call 'registernew' before any other phase advancement. This make 'changegroup.apply' register correct phase movement for the added and bundled nodes.

convert: use the new 'phase.registernew' function

localrepo: use the 'registernew' function to set the phase of new commit

phases: add a 'registernew' method to set new phases This new function will be used by code that adds new changesets. It ajusts the phase boundary to make sure added changesets are at least in their target phase (they end up in an higher phase if their parents are in a higher phase). Having a dedicated function also simplify the phases tracking. All the new nodes are passed as argument, so we know that all of them needs to have their new phase registered. We also know that no other nodes will be affected, so no extra computation are needed. This function differ from 'retractboundary' where some nodes might change phase while some other might not. It can also affect nodes not passed as parameters. These simplification also apply to the computation itself. For now we use '_retractboundary' there by convenience, but we may introduces simpler code later. While registering new revisions, we still need to check the actual phases of the added node because it might be higher than the target phase (eg: target is draft but parent is secret). We will migrate users over the next changesets.

phases: extract the core of boundary retraction in '_retractboundary' At the moment the 'retractboundary' function is called for multiple reasons: First, actually retracting boundaries. There are only two cases for theses: 'hg phase --force' and 'hg qimport'. This will need extra graph computation to retrieve the phase changes. Second, setting the phases of newly added changesets. In this case we already know all the affected nodes and we just needs to register different information (old phase is None). Third, when reducing the set of roots when advancing phase. The phase are already properly tracked so we do not needs anything else in this case. To deal with this difference in phase tracking, we extract the core logic into a private method that all three cases can use.

phases: track phase movements in 'advanceboundary' Makes advanceboundary record the phase movement of affected revisions in tr.changes['phases']. The tracking is not usable yet because the 'retractboundary' function can also affect phases. We'll improve that in the coming changesets.

phases: extract the intermediate set of affected revs When advancing phases, we compute the new roots for the phases above. During this process, we need to compute all the revisions that change phases (to the new target phases). Extract these revisions into a separate variable. This will be useful to record the phase changes in the transaction.

phase: put retractboundary out of the loop in advanceboundary It seems that we were calling retractboundary for each phases to process. Putting the retractboundary out of the loop reduce the number of calls, helping tracking the phases changes.

match: make unionmatcher a proper matcher unionmatcher is currently used where only a limited subset of its functions will be called. Specifically, visitdir() is never called. The next patch will pass it to dirstate.walk() where it will matter that visitdir() is correctly implemented, so let's fix that. Also add the explicitdir etc that will also be assumed by dirstate.walk() to exist on a matcher. Differential Revision: https://phab.mercurial-scm.org/D58

match: write forceincludematcher using unionmatcher The forceincludematcher is simply a unionmatcher of a includematcher (matching paths recursively) with the given matcher. Since the forceincludematcher is only used by sparse, move it there. I don't have a good sparse repo setup to test performance impact on. Differential Revision: https://phab.mercurial-scm.org/D57

histedit: extract InterventionRequired transaction handling to utils rebase will have similar logic, so let's extract it. Besides, it makes the histedit code more readable. We may want to parametrize acceptintervention() by the exception(s) that should result in transaction close. Differential Revision: https://phab.mercurial-scm.org/D66

histedit: create transaction outside of try Just a little refactoring to simplify the next patch. Differential Revision: https://phab.mercurial-scm.org/D65

histedit: remove transaction from state object The transaction is only used within a single function, so no need to store it on the state object. Differential Revision: https://phab.mercurial-scm.org/D64

phabricator: verify local tags before trusting them Previously we trust local tags blindly and that could cause wrong Differential Revision to be updated, when people switch between Phabricator instances. This patch adds verification logic to detect such issue and remove problematic tags. For example, a tag "D19" was on node "X", the code will fetch all diffs attached to D19, and check if nodes server-side overlaps with nodes in precursors. If they do not overlap, create a new Differential Revision. Test Plan: Use a test Phabricator instance, send patches using `hg phabsend`, then change the local tag manually to a wrong Differential Revision number. Amend the patch and send again. Make sure the tag gets ignored and deleted. Differential Revision: https://phab.mercurial-scm.org/D36

phabricator: finding old nodes in batch This allows us to do extra sanity checks using batch APIs to prevent updating a wrong revision, which could happen when people switch Phabricator instances and having stale tags living in the repo. Differential Revision: https://phab.mercurial-scm.org/D34

phabricator: respect metadata sent by arc Previously we only respect hg:meta sent by phabsend. This patch makes it respect local:commits sent by arc as well. This avoids issues where phabread could lose the author information. Test Plan: Commit using a customized user, send the patch using arc to a test Phabricator instance, and then read the patch using phabread. Make sure it preserves the user information. Differential Revision: https://phab.mercurial-scm.org/D33

dirstate: update backup functions to take full backup filename Update the dirstate functions so that the caller supplies the full backup filename rather than just a prefix and suffix. The localrepo code was already hard-coding the fact that the backup name must be (exactly prefix + "dirstate" + suffix): it relied on this in _journalfiles() and undofiles(). Making the caller responsible for specifying the full backup name removes the need for the localrepo code to assume that dirstate._filename is always "dirstate". Differential Revision: https://phab.mercurial-scm.org/D68

util: remove unused ctxmanager This was meant as a substitute for Python's "with" with multiple context managers before we moved to Python 2.7. We're now on 2.7, so we should have no reason to keep ctxmanager. "hg grep --all ctxmanager" says that it was never used anyway. Differential Revision: https://phab.mercurial-scm.org/D73

codemod: simplify nested withs This is the result of running: python codemod_nestedwith.py **/*.py where codemod_nestedwith.py looks like this: #!/usr/bin/env python # codemod_nestedwith.py - codemod tool to rewrite nested with # # Copyright 2017 Facebook, Inc. # # This software may be used and distributed according to the terms of the # GNU General Public License version 2 or any later version. from __future__ import absolute_import, print_function import sys import redbaron def readpath(path): with open(path) as f: return f.read() def writepath(path, content): with open(path, 'w') as f: f.write(content) def main(argv): if not argv: print('Usage: codemod_nestedwith.py FILES') for i, path in enumerate(argv): print('(%d/%d) scanning %s' % (i + 1, len(argv), path)) changed = False red = redbaron.RedBaron(readpath(path)) processed = set() for node in red.find_all('with'): if node in processed or node.type != 'with': continue top = node child = top[0] while True: if len(top) > 1 or child.type != 'with': break # estimate line length after merging two "with"s new = '%swith %s:' % (top.indentation, top.contexts.dumps()) new += ', %s' % child.contexts.dumps() # only do the rewrite if the end result is within 80 chars if len(new) > 80: break processed.add(child) top.contexts.extend(child.contexts) top.value = child.value top.value.decrease_indentation(4) child = child[0] changed = True if changed: print('updating %s' % path) writepath(path, red.dumps()) if __name__ == "__main__": sys.exit(main(sys.argv[1:])) Differential Revision: https://phab.mercurial-scm.org/D77

reposvfs: add a ward to check if locks are properly taken we wrap 'repo.svfs.audit' to check for the store lock when accessing file in '.hg/store' for writing. This caught a couple of instance where the transaction was released after the lock, we should probably have a dedicated checker for that case.

repovfs: add a ward to check if locks are properly taken When the appropriate developer warnings are enabled, We wrap 'repo.vfs.audit' to check for locks when accessing file in '.hg' for writing. Another changeset will add a 'ward' for the store vfs (svfs). This check system has caught a handful of locking issues that have been fixed in previous series (mostly in 4.0). I expect another batch to be caught in third party extensions. We introduce two real exceptions from extensions 'blackbox.log' (because a lot of read-only operations add entry to it), and 'last-email.txt' (because 'hg email' is currently a read only operation and there is value to keep it this way). In addition we are currently allowing bisect to operate outside of the lock because the current code is a bit hard to get properly locked for now. Multiple clean up have been made but there is still a couple of them to do and the freeze is coming.

vfs: allow to pass more argument to audit We want to be able to do more precise check when auditing a path depending of the intend of the file access (eg read versus write). So we now pass the 'mode' value to 'audit' and update the audit function to accept them. This will be put to use in the next changeset.

zsh_completion: add '--partial' flag to completions for 'import' (issue5618) When using the 'import' command running ZSH together with its Mercurial-specific completions the flag '--partial' (introduced in Mercurial 3.1, 2014-08-01) is not offered as completion option. This patch adds it to the list of completions for the 'import' command.

configitems: register the 'bugzilla.password' config

configitems: register the 'bugzilla.mocklog' config

test-amend: fix HGEDITOR shell script to run on Windows Windows doesn't know how to launch *.sh directly. This workaround is used in several other tests.

test-paths: mask out JSON specific path differences for Windows The backslashes in the local paths were being escaped with another backslash, and $TESTTMP doesn't match against the double backslashed path. This doesn't happen without the 'json' filter.

tests: categorize entries in blacklist for fsmonitor for future reference

tests: avoid tests related to directory permission if running with fsmonitor

tests: add extra output for fsmonitor at checking under .hg

tests: add extra output for fsmonitor at showconfig

tests: take extra care for fsmonitor at enabling incompatible extension This is worthwhile, because these tests don't focus mainly on using incompatible extensions itself.

tests: remove meaningless entries from blacklist for fsmonitor IMHO, these tests were listed up in blacklist at 3b67f27bb908, because fsmonitor extension wasn't correctly installed before actual testing at that revision (this issue was fixed by 776fd2e2cf5a).

tests: remove meaningless comment from blacklist for fsmonitor These tests can run successfully since fe0667cc521e.

test-https: stabilize for Windows A leading '/' without a following drive letter is resolved to the location of the MSYS installation. These could be collapsed into a single line of output, but this seems more self documenting.

tagmerge: use workingfilectx to write merged tags This function already does an excellent job of reading from context objects; we simply need to change the single write call to eliminate all uses of the wvfs. As with past changes, the effect should be a no-op but opens the door to in-memory merge later by using different context objects.

run-tests: add color to output if pygments is available The output of run-tests has no formatting by default, which hampers readability. This patch colors the diff output when pygments is available. To avoid coloring even when pygments is available, use --color never.

win32: work around a WinError problem handling HRESULT types I ran into this ctypes bug while working with the Crypto API. While this could be an issue with any Win32 API in theory, the handful of things that we call are older functions that are unlikely to return COM errors, so I didn't retrofit this everywhere.

test-check-pyflakes: do not leave test.py in the repo Differential Revision: https://phab.mercurial-scm.org/D67

revset: add experimental ancestors/descendants relation subscript The relation name is 'generations' now, which may be changed in future.

revset: add experimental relation and subscript operators The proposed syntax [1] was originally 'set{n rel}', but it seemed slightly confusing if template is involved. On the other hand, we want to keep 'set[n]' for future extension. So this patch introduces 'set#rel[n]' ternary operator. I chose '#' just because it looks like applying an attribute. This also adds stubs for 'set[n]' and 'set#rel' operators since these syntax elements are fundamental for constructing 'set#rel[n]'. [1]: https://www.mercurial-scm.org/wiki/RevsetOperatorPlan#ideas_from_mpm

revset: do not compute weight for integer literal argument In x^n and x~n, n isn't a set expression. There's no need to optimize the right-hand side.

templatekw: export ui.paths as {peerpaths} It's sometimes useful to show hyperlinks in log output. "{get(peerpaths, "default")}/rev/{node}" Since each path may have sub options, "{peerpaths}" is structured as a dict of dicts, but the inner dict is rendered as if it were a string URL. The implementation is ad-hoc, so there are some weird behaviors described in the test. We might need to introduce a proper way of handling a hybrid scalar object. This patch adds _hybrid.__getitem__() so d['path']['url'] works. The keyword is named as "peerpaths" since "paths" seemed too generic in log context.

summary: fix type of empty unresolved list It was okay because tested as a boolean prior to calling len(), but looked incorrect.

vfs: rename auditvfs to proxyvfs Since we've removed mustaudit property, auditvfs has no auditing business. It's just a utility class for vfs wrappers.

streamclone: comment why path auditing is disabled in generatev1() Copied from 39c6e349dfff. I wasn't sure whether it's for optimization or suppressing unwanted error.

streamclone: close large revlog files explicitly in generatev1()

bundle2: no longer use 'retractboundary' in updatephases The new 'phase-heads' forced all added node to secret before advancing the boundary to work around the fact changesets were added as draft by default. This is no longer necessary since the changegroup part can now use the 'targetphase' parameter. Not doing this retract boundary call has a couple of advantages: * This makes implementing phases change tracking in the transaction much simpler since retract boundary can become a rare case. * Bundling secret changesets is not the norm. Exchange never does that and even for strip, the use-case is not common.Skipping the retract boundary will avoid useless work here. * Sending phase update on push can be simplified since we can rely on the behavior of 'cg.apply' for most of it. This means less phases update send for example. * We no longer needs to track and use the addednodes during unbundling. This make it possible to have multiple 'changegroup' and 'phase-heads' parts in the same bundle without them interfering with each others. The new part has not been part of any release yet so we do not offer backward compatibility yet. It is important to update this semantic before the 4.3 freeze happens.

bundle2: automatically add 'targetphase' parameter in writenewbundle If we are bundling secret changeset and the bundle will contain phase, we request the changegroup to be applied as secret. It will be useful for next patch as we are now sure that secrets changesets are applied as secret and not applied as draft then forced to secret.

bundle2: support the 'targetphase' parameter for the changegroup part By default unbundled changesets are drafts. We want to reduce the number of phases changes during unbundling by giving the possibility to the bundle to indicate the phase of unbundled changesets. The longer terms goal is to add phase movement tracking in tr.changes and the 'retractboundary' call is making it more complicated than we want.

changegroup: stop treating strip as special when dealing with phases Since 8e3021fd1a44, the strip bundle includes the phases of the stripping node. Hence we don't need this special case anymore. Dropping it will helps make the phase behavior more consistent across all exchanges medium.

match: inverse _anypats(), making it _prefix()

amend: new extension providing the amend command Various third parties have implemented the `amend` command, which is in high demand. This patch adds it as an experimental extension so its interface could be formalized in core directly. Since `commit --amend` is basically what `amend` should do. The command is just a thin wrapper around `commit --amend` and just prevent the editor from popping up by passing `--message`.

py3: make localrepo filtered repo cache work on py3 I don't know if this is the right fix, but it makes test-py3-commands.t pass again. Differential Revision: https://phab.mercurial-scm.org/D56

configitems: register the 'bugzilla.host' config

configitems: register the 'bugzilla.fixstatus' config

configitems: register the 'bugzilla.fixresolution' config

configitems: register the 'bugzilla.fixregexp' config

configitems: register the 'bugzilla.db' config

configitems: register the 'bugzilla.bzuser' config

configitems: register the 'bugzilla.bzurl' config

configitems: register the 'bugzilla.bzemail' config

configitems: register the 'bugzilla.bzdir' config

configitems: register the 'bugzilla.apikey' config

revlog: use struct.Struct instances for slight performance wins Differential Revision: https://phab.mercurial-scm.org/D32

revlog: micro-optimize the computation of hashes Differential Revision: https://phab.mercurial-scm.org/D31

hgweb: re-implement followlines UI selection using buttons This changeset attempts to solve two issues with the "followlines" UI in hgweb. First the "followlines" action is currently not easily discoverable (one has to hover on a line for some time, wait for the invite message to appear and then perform some action). Second, it gets in the way of natural line selection, especially in filerevision view. This changeset introduces an additional markup element (a <button class="btn-followlines">) alongside each content line of the view. This button now holds events for line selection that were previously plugged onto content lines directly. Consequently, there's no more action on content lines, hence restoring the "natural line selection" behavior (solving the second problem). These buttons are hidden by default and get displayed upon hover of content lines; then upon hover of a button itself, a text inviting followlines section shows up. This solves the first problem (discoverability) as we now have a clear visual element indicating that "some action could be perform" (i.e. a button) and that is self-documented. In followlines.js, all event listeners are now attached to these <button> elements. The custom "floating tooltip" element is dropped as <button> elements are now self-documented through a "title" attribute that changes depending on preceding actions (selection started or not, in particular). The new <button> element is inserted in followlines.js script (thus only visible if JavaScript is activated); it contains a "+" and "-" with a "diff-semantics" style; upon hover, it scales up. To find the parent element under which to insert the <button> we either rely on the "data-selectabletag" attribute (which defines the HTML tag of children of class="sourcelines" element e.g. <span> for filerevision view and <tr> for annotate view) or use a child of the latter elements if we find an element with class="followlines-btn-parent" (useful for annotate view, for which we have to find the <td> in which to insert the <button>). On noticeable change in CSS concerns the "margin-left" of span:before pseudo-elements in filelog view that has been increased a bit in order to leave space for the new button to appear between line number column and line content one. Also note the "z-index" addition for "annotate-info" box so that the latter appears on top of new buttons (instead of getting hidden). In some respect, the UI similar to line commenting feature that is implemented in popular code hosting site like GitHub, BitBucket or Kallithea.

localrepo: cache types for filtered repos (issue5043) Python introduces a reference cycle on dynamically created types via __mro__, making them very easy to leak. See https://bugs.python.org/issue17950. Previously, repo.filtered() created a type on every invocation. Long-running processes (like `hg convert`) could call this function thousands of times, leading to a steady memory leak. Since we're Unable to stop the leak because this is a bug in Python, the next best thing is to contain it. This patch adds a cache of of the dynamically generated repoview/filter types on the localrepo object. Since we only generate each type once, we cap the amount of memory that can leak to something reasonable. After this change, `hg convert` no longer leaks memory on every revision. The process will likely grow memory usage over time due to e.g. larger manifests. But there are no leaks.

convert: transcode CVS log messages by specified encoding (issue5597) Converting from CVS to Mercurial assumes that CVS log messages in "cvs rlog" output are encoded in UTF-8 (or basic Latin-1). But cvs itself is usually unaware of encoding of log messages, in practice. Therefore, if there are commits, of which log message is encoded in other than UTF-8, log message of corresponded revisions in the converted repository will be broken. To avoid such broken log messages, this patch transcodes CVS log messages by encoding specified via "convert.cvsps.logencoding" configuration. This patch accepts multiple encoding for convenience, because "multiple encoding mixed in a repository" easily occurs. For example, UTF-8 (recent POSIX), cp932 (Windows), and EUC-JP (legacy POSIX) are well known encoding for Japanese.

fsmonitor: execute setup procedures only if dirstate is already instantiated Before this patch, reposetup() of fsmonitor executes setup procedures for dirstate, even if it isn't yet instantiated at that time. On the other hand, dirstate might be already instantiated before reposetup() intentionally (prefilling by chg, for example, see bf3af0eced44 for detail). If so, just discarding already instantiated one in reposetup() causes issue. To resolve both issues above, this patch executes setup procedures, only if dirstate is already instantiated. BTW, this patch removes "del repo.unfiltered().__dict__['dirstate']", because it is responsibility of the code path, which causes instantiation of dirstate before reposetup(). After this patch, using localrepo.isfilecached() should avoid creating the corresponded entry in repo.unfiltered().__dict__.

fsmonitor: centralize setup procedures for dirstate

fsmonitor: avoid needless instantiation of dirstate Using repo.local() instead of util.safehasattr(repo, 'dirstate') also avoids executing setup procedures for remote repository (including statichttprepo). This is reason why this patch also removes a part of subsequent comment, and try/except for AttributeError at accessing to repo.wvfs.

journal: use wrapfilecache instead of wrapfunction on func of filecache wrapfilecache() on filecache-ed property works more strictly than wrapfunction() directly on func() of filecache.

journal: execute setup procedures for already instantiated dirstate If dirstate is instantiated before reposetup() of journal extension, it doesn't have "journalstorage" property, even if it is instantiated via wrapdirstate() wrapping repo.dirstate(), because wrapdirstate() works as same as original one before marking repo as "journal"-ing in reposetup(). This issue can be reproduced by running test-journal.t or test-journal-share.t with fsmonitor-run-tests.py. On the other hand, just discarding already instantiated dirstate in reposetup() prevents chg from filling dirstate before reposetup() (see bf3af0eced44 for detail). Therefore, this patch executes setup procedures for already instantiated dirstate explicitly in reposetup(). To centralize setup procedures for dirstate, this patch also factors them out from wrapdirstate().

localrepo: add isfilecached to check filecache-ed property is already cached isfilecached() encapsulates internal implementation of filecache-ed property. "name in repo.unfiltered().__dict__" or so can't be used for this purpose, because corresponded entry in __dict__ might be discarded by repo.invalidate(), repo.invalidatedirstate() or so (fsmonitor does so, for example). This patch makes isfilecached() return not only whether filecache-ed property is already cached, but also already cached value (or None), in order to avoid subsequent access to cached object via "repo.NAME", which prevents main Mercurial procedure after reposetup() from validating cache.

sslutil: check for missing certificate and key files (issue5598) Currently, sslutil._hostsettings() performs validation that web.cacerts exists. However, client certificates are passed in to the function and not all callers may validate them. This includes httpconnection.readauthforuri(), which loads the [auth] section. If a missing file is specified, the ssl module will raise a generic IOException. And, it doesn't even give us the courtesy of telling us which file is missing! Mercurial then prints a generic "abort: No such file or directory" (or similar) error, leaving users to scratch their head as to what file is missing. This commit introduces explicit validation of all paths passed as arguments to wrapsocket() and wrapserversocket(). Any missing file is alerted about explicitly. We should probably catch missing files earlier - as part of loading the [auth] section. However, I think the sslutil functions should check for file presence regardless of what callers do because that's the only way to be sure that missing files are always detected.

match: override matchfn instead of __call__ for consistency The matchers that were recently moved into core from the sparse extension override __call__, while the previously existing matchers override matchfn. Let's switch to the latter for consistency.

match: express anypats(), not prefix(), in terms of the others When I added prefix() in 9789b4a7c595 (match: introduce boolean prefix() method, 2014-10-28), we already had always(), isexact(), and anypats(), so it made sense to write it in terms of them (a prefix matcher is one that isn't any of the other types). It's only now that I realize that it's much more natural to define prefix() explicitly (it's one that uses path: patterns, roughly speaking) and let anypats() be defined in terms of the others. Remember that these methods are all used for determining which fast paths are possible. anypats() simply means that no fast paths are possible (it could be called complex() instead). Further evidence is that rootfilesin:some/dir does not have any patterns, but it's still considered to be an anypats() matcher. That's because anypats() really just means that it's not a prefix() matcher (and not always() and not isexact()). This patch thus changes prefix() to return False by default and anypats() to return True only if the other three are False. Having anypats() be True by default also seems like a good thing, because it means forgetting to override it will lead only to performance bugs, not correctness bugs. Since the base class's implementation changes, we're also forced to update the subclasses. That change exposed and fixed a bug in the differencematcher: for example when both its two input matchers were prefix matchers, we would say that the result was also a prefix matcher, which is incorrect, because e.g "path:dir - path:dir/foo" no longer matches everything under "dir" (which is what prefix() means).

match: make nevermatcher an exact matcher and a prefix matcher The m.isexact() and m.prefix() methods are used by callers to determine whether m.files() can be used for fast paths. It seems safe to let callers to any fast paths it can that rely on the empty m.files().

revset: define successors revset This revset returns all successors, including transit nodes and the source nodes (to be consistent with existing revsets like "ancestors"). To filter out transit nodes, use `successors(X)-obsolete()`. To filter out divergent case, use `successors(X)-divergent()-obsolete()`. The revset could be useful to define rebase destination, like: `max(successors(BASE)-divergent()-obsolete())`. The `max` is to deal with splits. There are other implementations where `successors` returns just one level of successors, and `allsuccessors` returns everything. I think `successors` returning all successors by default is more user friendly. We have seen cases in production where people use 1-level `successors` while they really want `allsuccessors`. So it seems better to just have one single revset returning all successors by default to avoid user errors. In the future we might want to add `depth` keyword argument to it and for other revsets like `ancestors` etc. Or even build some flexible indexing syntax [1] to satisfy people having the depth limit requirement. [1]: https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-July/101140.html

sparse: shorten try..except block in updateconfig() It now only covers refreshwdir(). This is what importfromfiles() does. I think it is the more appropriate behavior.

sparse: clean up updateconfig() * Use context manager for wlock * Rename oldsparsematch to oldmatcher * Always call parseconfig() because parsing an empty string yields the same result as the old code

sparse: move config updating function into core As part of the move, the ui argument was dropped. Additional fixups will be made in a follow-up commit.

dirstate: expose a sparse matcher on dirstate (API) The sparse extension performs a lot of monkeypatching of dirstate to make it sparse aware. Essentially, various operations need to take the active sparse config into account. They do this by obtaining a matcher representing the sparse config and filtering paths through it. The monkeypatching is done by stuffing a reference to a repo on dirstate and calling sparse.matcher() (which takes a repo instance) during each function call. The reason this function takes a repo instance is because resolving the sparse config may require resolving file contents from filelogs, and that requires a repo. (If the current sparse config references "profile" files, the contents of those files from the dirstate's parent revisions is resolved.) I seem to recall people having strong opinions that the dirstate object not have a reference to a repo. So copying what the sparse extension does probably won't fly in core. Plus, the dirstate modifications shouldn't require a full repo: they only need a matcher. So there's no good reason to stuff a reference to the repo in dirstate. This commit exposes a sparse matcher to dirstate via a property that when looked up will call a function that eventually calls sparse.matcher(). The repo instance is bound in a closure, so it isn't exposed to dirstate. This approach is functionally similar to what the sparse extension does today, except it hides the repo instance from dirstate. The approach is not optimal because we have to call a proxy function and sparse.matcher() on every property lookup. There is room to cache the matcher instance in dirstate. After all, the matcher only changes if the dirstate's parents change or if the sparse config changes. It feels like we should be able to detect both events and update the matcher when this occurs. But for now we preserve the existing semantics so we can move the dirstate sparseness bits into core. Once in core, refactoring becomes a bit easier since it will be clearer how all these components interact. The sparse extension has been updated to use the new property. Because all references to the repo on dirstate have been removed, the code for setting it has been removed.

sparse: use self instead of repo.dirstate "self" here is the dirstate instance. I'm pretty confident that self and repo.dirstate will be the exact same object. So remove a dependency on repo by just looking at self.

sparse: move code for importing rules from files into core This is a pretty straightforward port. Some code cleanup was performed. But no major changes to the logic were made. I'm not a huge fan of this function because it does multiple things. I'd like to get things into core first to facilitate refactoring later. Please also note the added inline comment about the oddities of writeconfig() and the try..except to undo it. This is because of the hackiness in which the sparse matcher is obtained by various consumers, notably dirstate. We'll need a massive refactor to address this. That refactor is effectively blocked on having the sparse dirstate hacks live in core.

sparse: refactor activeprofiles into a generic function (API) activeprofiles() is a special case of a more generic function. Furthermore, that generic function is essentially already implemented inline in the sparse extension. So, refactor activeprofiles() to a generic activeconfig(). Change the only consumer of activeprofiles() to use it. And have the inline implementation in the sparse extension use it.

check-code: prohibit `if False` antipattern Differential Revision: https://phab.mercurial-scm.org/D20

convert: remove `if False` block This code has never run since its introduction on July 18th, 2007. It's time for it to go. Differential Revision: https://phab.mercurial-scm.org/D19

filterpyflakes: move self-test into test file This will avoid a false positive on an upcoming check-code rule. Differential Revision: https://phab.mercurial-scm.org/D18

test-subrepo: demonstrate a status problem when merge deletes a file At the interactive update prompt, if (c) is chosen and then followed by `hg rm`, both `status -R` and `status -S` show the file as 'R', and `files -R` shows no files (OK, because explicitly removed files aren't supposed to be listed). If `rm` follows selecting (c), then both flavors of `status` list the file as '!', and `files -R` lists the missing file. So somehow, the (d) option has followed a third path.

subrepo: make the output references to subrepositories consistent Well, mostly. The annotation on subrepo functions tacks on a parenthetical to the abort message, which seems reasonable for a generic mechanism. But now all messages consistently spell out 'subrepository', and double quote the name of the repo. I noticed the inconsistency in the change for the last commit.

subrepo: consider the parent repo dirty when a file is missing This simply passes the 'missing' argument down from the context of the parent repo, so the same rules apply. subrepo.bailifchanged() is hardcoded to care about missing files, because cmdutil.bailifchanged() is too. In the end, it looks like this addresses inconsistencies with 'archive', 'identify', blackbox logs, 'merge', and 'update --check'. I wasn't sure how to implement this in git, so that's left for someone more familiar with it.

archival: flag missing files as a dirty wdir() in the metadata file (BC) Since the identify command adds a '+' for missing files, it's reasonable that this does too. Perhaps the node field's hex value should be p1+p2 for merges?

cmdutil: simplify the dirty check in howtocontinue() This is equivalent to the previous code. But it seems to me that if the user is going to be prompted that a commit is needed, missing files should be ignored, but branch and merge changes shouldn't be.

blackbox: simplify the dirty check Same idea (and possibly incorrect behavior) as the previous commit.

identify: simplify the dirty check This is equivalent to the previous code, but it seems better to be explicit about what aspects of dirty are being ignored. Perhaps they shouldn't be, since the help text says 'followed by a "+" if the working directory has uncommitted changes'. Both merges and branch changes are committable, even if the files are unchanged. Additionally, this will make the `identify` command notice missing subrepo files, once subrepos are taught to look for missing files.

tests: tweak the subrepo dirty state tests This is a continuation of 439b4d005b4a. I overlooked that blackbox logs also have a dirty marker. Also, the `hg update --check` test was updating to a revision where the deleted file wasn't tracked, which is why status seemed to show the deleted file was restored.

match: combine regex code for path: and relpath: The regexes for path: and relpath: patterns are the same (since the paths have already been normalized at the point we create the regexes). I don't think the "if pat == '.'" will have any effect relpath: because relpath: patterns will have the root directory already normalized to '' by pathutil.canonpath() (unlike path:, for which the root gets normalized to '.' by util.normpath()).

match: remove unnecessary '^' from regexes The regexes are passed to re.match(), which matches against the beginning of the input, so the '^' doesn't do anything. Note that unrooted patterns, such as globs and regexes from .hgignore are instead achieved by adding '.*' to the expression given by the user. (That's unless the user's expression started with '^', in which case the '.*' is not added, perhaps to keep the regex cleaner?)

sparse: access status fields by name instead of deconstructing it The status tuples has had named fields for a few years now.

sparse: move printing of sparse config changes function into core As part of the port, all arguments now have default values of 0. Strings are now also given the i18n treatment.

sparse: move code for clearing rules to core This is a pretty straightforward port.

sparse: move post commit actions into core Instead of wrapping committablectx.markcommitted(), we inline the call into workingctx.markcommitted(). Per smf's review, workingctx is the proper location for this code, as committablectx is the shared base class for it and memctx. Since this code touches the working directory, it belongs in workingctx.

cleanupnode: do not use generator for node mapping The 'successors' part of the mappings used of be a tuple. This avoid issue from code consuming the generator "by mistake". For example, an extension inspecting the mapping content used to be able to iterate over the successors mapping without consequence. Since the mapping are small we do not expect any performance impact we use tuple again for this.

histedit: use scmutil.cleanupnodes (BC) This is marked as BC because the strip backup file name has changed.

histedit: unify strip backup files on success (BC) Previously we wrote two different strip backup files on success. This patch unifies them. It will make scmutil.cleanupnodes migration more smooth.

histedit: pass multiple nodes to strip (BC) Previously, histedit.cleanupnode pass root nodes one by one. Since repair.strip takes multiple nodes and can handle them just fine, pass all strip roots at once. This is BC because the number of strip backup files may change from N to 1.

histedit: remove "name" parameter from cleanupnode functions The "name" parameter is not used any longer so let's remove it.

histedit: remove "should strip" debug message The debug message was not used anywhere. Removed it to make scmutil.cleanupnodes migration easier to reason about.

histedit: move topmost bookmark movement to a separate function histedit treats topmost bookmark movement specially. The rest of the bookmark movement could be handled by scmutil.cleanupnodes. So let's move the special logic out to make the patch easier to review.

histedit: remove moving bookmarks message on verbose (BC) This is more consistent with other commands, like "commit -v" won't show bookmark movement messages. It will make migrating to scmutil.cleanupnodes easier.

test-largefiles-update: conditionalize output instead of tests

test-status-rev: conditionalize output instead of tests

test-mq: conditionalize output instead of tests

test-annotate: conditionalize output instead of tests

test-addremove: conditionalize output instead of tests

test-tools: conditionalize output instead of tests

test-rebase: conditionalize output instead of tests This should help prevent breakage like was fixed in the last patch.

tests: stabilize on Windows I'm not sure if the difference on Windows for test-sparse.t is expected or not. It looks like unless the leading '/' is followed by a drive letter, '/' is resolved to 'C:/MinGW/msys/1.0'. But both cases abort with "not under root" instead of just warning.

revset: make repo.anyrevs accept customized alias override (API) Previously repo.anyrevs only expand aliases in [revsetalias] config. This patch makes it more flexible to accept a customized dict defining aliases without having to couple with ui. revsetlang.expandaliases now has the signature (tree, aliases, warn=None) which is more consistent with templater.expandaliases. revsetlang.py is now free from "ui", which seems to be a good thing.

tests: quote $PYTHON for Windows When unquoted, MSYS sees the colon between the drive letter and path as a Unix path separator and unhelpfully splits on it, feeding only the drive letter as the command. Much chaos ensues. I vaguely remember trying to get the test runner to use /letter/path/to/exe syntax the last time this happened, without success. I doubt a check-code rule would work, since sometimes it is quoted, and sometimes the quotes are escaped.

amend: use scmutil.cleanupnodes (BC) This is marked as BC because the strip backup file name has changed.

rebase: remove "if True" The "if True" block was to make the last patch easier to review. This patch removes "if True" and unindents the block.

rebase: use scmutil.cleanupnodes (issue5606) (BC) This patch migrates rebase to use scmutil.cleanupnodes API. It simplifies the code and makes rebase code reusable inside a transaction. This is a BC because the backup file is no longer strip-backup/*-backup.hg, but strip-backup/*-rebase.hg. The latter looks more reasonable since the directory name is "strip-backup" so there is no need to repeat "backup". I think the backup file name change is probably fine as a BC, since we have changed it before (aa4a1672583e) and didn't get complains. The end result of this series will be a much more consistent and unified backup names: command | old backup file suffix | new backup file suffix ------------------------------------------------------------------- amend | amend-backup.hg | amend.hg histedit | backup.hg (could be 2 files) | histedit.hg (single file) rebase | backup.hg | rebase.hg strip | backup.hg | backup.hg (note: backup files are under .hg/strip-backup) It also fixes issue5606 as a side effect because the new "delayedstrip" code path will carefully examine nodes (safestriproots) to make sure orphaned changesets won't get stripped by accident. Some warning messages are changed to the new "warning: orphaned descendants detected, not stripping HASHES", which provides more information about exactly what changesets are left behind. Another minor behavior change is when there is an obsoleted changeset with a successor in the destination branch, bookmarks pointing to that obsoleted changeset will not be moved. I have commented in test-rebase-obsolete.t explaining why that is more desirable.

scmutil: make cleanupnodes delete divergent bookmarks cleanupnodes takes care of bookmark movement, and bookmark movement could cause bookmark divergent resolution as a side effect. This patch adds such bookmark divergent resolution logic so future rebase migration will be easier. The revset is carefully written to be equivalent to what rebase does today. Although I think it might make sense to remove divergent bookmarks more aggressively, for example: F book@1 | E book@2 | | D book | | | C |/ B book@3 | A When rebase -s C -d E, "book@1" will be removed, "book@3" will be kept, and the end result is: D book | C | F | E book@2 (?) | B book@3 | A The question is should we keep book@2? The current logic keeps it. If we choose not to (makes some sense to me), the "deleterevs" revset could be simplified to "newnode % oldnode". For now, I just make it compatible with the existing behavior. If we want to make the "deleterevs" revset simpler, we can always do it in the future.

scmutil: make cleanupnodes handle filtered node In some valid usecases, the "mapping" received by scmutil.cleanupnodes have filtered nodes. Use unfiltered repo to access them correctly. The added test case will fail with the old cleanupnodes code. This is important to migrate histedit to use the cleanupnodes API.

configitems: add alias support in config Aliases define optional alternatives to existing options. For example the old option ui.user was deprecated and replaced by ui.username. With this mechanism, it's even possible to create an alias to an option in a different section. Add ui.user as alias to ui.username as an example of this concept. The old alternates principle in ui.config is removed as it was used only for this option.

hgweb: use ui._unset to prevent a warning in configitems

dispatch: fix typo suggestion for disabled extension If the matching command lives in an in-tree extension (which is all we scan for), and the user has disabled that extension with "extensions.<name>=!", we were not finding it, because the path in _disabledextensions was the empty string. If the user had set "extensions.<name>=!<valid path>" it would work, so it seems like just a mistake that it didn't work.

tests: add tests for typoed commands This includes one test showing how disabling a command with e.g. "extensions.rebase=!" results in the command not being suggested. We'll fix that next.

sparse: inline signature cache clearing It is a trivial one-liner. No need to have a separate function.

sparse: move working directory refreshing into core This is a pretty straightforward move of the code. I converted the "force" argument to a keyword argument. Like other recent changes, this code is tightly coupled with working directory update code in merge.py. I suspect the code will become more tightly coupled over time, possibly even moved to merge.py. For now, let's get the code in core.

sparse: refactor update actions filtering and call from core merge.calculateupdates() now filters the update actions through sparse by default. The filtering no-ops if sparse isn't enabled or no sparse config is defined. The function has been refactored to behave more like a filter instead of a wrapper of merge.calculateupdates(). We should arguably take sparse into account earlier in merge.calculateupdates(). This patch preserves the old behavior of applying sparse at the end of update calculation, which is the simplest and safest approach.

sparse: move update action filtering into core This is a relatively straight port of the function. It is pretty large. So refactoring will be postponed to a subsequent commit.

sparse: move pruning of temporary includes into core This was our last method on the custom repo type, meaning we could remove that custom type and inline the 2 lines of code into reposetup(). As part of the move, instead of wrapping merge.update() from the sparse extension, we inline the function call. The ported function now no-ops if sparse isn't enabled, making it safe to always call. The call site in update() may not be the most appropriate. But it matches the previous behavior, which is the safest thing to do. It can be improved later.

sparse: move function for resolving sparse matcher into core As part of the move, the function arguments changed so revs are passed as a list instead of *args. This allows us to use keyword arguments properly. Since the plan is to integrate sparse into core and have it enabled by default, we need to prepare for a sparse matcher to always be obtained and operated on. As part of the move, we inserted code that returns an always matcher if sparse isn't enabled. Some callers in the sparse extension take this into account and conditionally perform matching depending on whether the special always matcher is seen. I /think/ this may have sped up some operations where the extension is installed but no sparse config is activated. One thing I'm ensure of in this code is whether os.path.dirname() is semantically correct. os.posixpath.dirname() (which is exported as pathutil.dirname) might be a better choise because all patterns should be using posix directory separators (/) instead of Windows (\). There's an inline comment that implies Windows was tested. So hopefully it won't be a problem. We can improve this in a follow-up. I've added a TODO to track it.

match: move matchers from sparse into core The sparse extension contains some matcher types that are generic and can exist in core. As part of the move, the classes now inherit from basematcher. always(), files(), and isexact() have been dropped because they match the default implementations in basematcher.

sparse: clean up config signature code Before, 0 was being used as the default signature value and we cast the int to a string. We also handled I/O exceptions manually. The new code uses cfs.tryread() so we always feed data into the hasher. The empty string does hash and and should be suitable for input into a cache key. The changes made the code simple enough that the separate checksum function could be inlined.

sparse: move config signature logic into core This is a pretty straightforward port. It will be cleaned up in a subsequent commit.

sparse: remove custom hash matcher With the recent change to always use repr(), this function was functionally identical to the version in fsmonitor it was replacing. So remove it.

sparse: override __repr__ in matchers sparse.py in FB's hg-experimental repo switched to using __repr__ for non-sparse matchers soon after hg core started overriding __repr__ in the matchers in match.py (because the core matchers also stopped having "includepat" and other attributes that sparse used to depend on). Let's finish that migration by implementing __repr__ in the sparse matchers as well. That also lets us remove the special handling of them in _hashmatcher().

tests: fix reference to undefined variable The delaypush() function had a reference to "repo" that was clearly supposed to be "pushop.repo". Instead of just fixing that, let's extract "pushop.repo.ui" to a variable, since that's the only piece of the repo that's needed in the function. I have not looked into why I saw a different result in the test to start with, but that's for another patch anyway.

shelve: don't reimplement mergestate.unresolved()

summary: don't reimplment mergestate.unresolved()

mergestate: implement unresolvedcount() in terms of unresolved() This simplifies the method slightly. It does create a full list of paths while doing so, but it's not a lot of data anyway (besides, I would think references to strings are no larger than (references to?) True).

mergestate: make unresolved() use iteritems() mergestate.unresolved() is a generator, so it seems better for it to rely on iteritems() than items(), although it also seems unlikely for it to make a noticeable difference.

changegroup: don't fail on empty changegroup (API) I don't know why applying an empty changegroup should be an error. It seems harmless. I suspect the check was there to find code that creates empty changegroups just because that would be wasteful. Let's use develwarn() for that instead, so we catch any such cases that run with our test runner, but we still allow others to generate empty changegroups if they want to. We have run into this check at Google once or twice and had to work around it, but I'm changing this not so much because of that, but because it seems like it shouldn't be an error. I also changed the message slightly to be more modern ("changelog group" -> "changegroup") and more generic ("received" -> "applied").

changegroup: remove option to allow empty changegroup (API) No caller sets the "emptyok" option, so let's remove it.

strip: don't allow empty changegroup in bundle1 Applying an empty changegroup has been an error since the beginning. The only exception was strip, which would allow to apply an empty changegroup from the temporary bundle. However, the emptyok=True option was only set for bundle1 bundles. In other words, temporary bundle2 bundles would fail if they were empty. Bundle2 has now been used enough that it seems safe to say that we simply don't create bundle2 bundles with empty changegroups. That also suggests that we never create bundle1 bundles with empty changegroups (i.e. empty bundle1 bundles, since bundle1 is just a changegroup), because, AFAICT, the code leading up to the application of the bundle is the same for bundle1 and bundle2. Therefore, let's stop passing emptyok=True, so we more clearly get the same behavior for bundle1 and bundle2.

match: minor cleanups to patternmatcher and includematcher The "patterns"/"include" in "patternspat"/"includepat" is redundant, so drop it. Also a "_" prefix since it's "private". Inline the "pm"/"im" variables.

py3: fix test-diff-newlines.t to be compatible with py3

sparse: move some temporary includes functions into core Functions for reading and writing the tempsparse file have been moved. prunetemporaryincludes() will be moved separately because it is non-trivial.

sparse: move config file writing into core The code was refactored during the move to be more procedural instead of using string formatting. This has the benefit of not writing empty sections, which changed tests.

localrepo: add sparse caches The sparse extension maintains caches for the sparse files to a signature and a signature to a matcher. This allows the sparse matchers to be resolved quickly, which is apparently something that can occur in loops. This patch ports the sparse caches to the localrepo class pretty much as-is. There is potentially room to improve the caching mechanism. But that can be done as a follow-up. The default invalidatecaches() now clears the relevant sparse cache. invalidatesignaturecache() has been moved to sparse.py.

sparse: move active profiles function into core Also includes some light formatting changes.

sparse: move resolving of sparse patterns for rev into core This method is reasonably well-contained and simple to move. As part of the move, some light formatting was performed. A "working copy" reference in an error message was changed to "working directory." The biggest change was to _refreshoncommit() in sparse.py. It was previously checking for the existence of an attribute on the repo instance. Since the moved function now returns empty data if sparse isn't enabled, we unconditionally call the new function. However, we do have to protect another method call in that function. This will all be unhacked eventually.

sparse: variable to track if sparse is enabled Currently, the sparse extension sniffs repo instances for attributes defined by the sparse extension to determine if sparse is enabled. As we move code away from repo instances, these checks will be a bit more brittle. We introduce a module-level variable to track whether sparse is enabled as a temporary workaround.

sparse: move profile reading into core One more step towards weaning off methods on repo instances and moving code to core. While this function is only used once and is simple, it needs to exist on its own so Facebook can monkeypatch it to enable simplecache integration.

sparse: move config parsing into core This patch marks the beginning of moving code from the sparse extension into core. The goal is to move as much of the functionality as possible into core, where it will be an experimental feature. The extension will likely continue to exist to enable the feature and provide UI elements. As part of the move, the repo method was converted to a module function. It doesn't need to exist on repos. An error message was also updated to reflect that an error isn't necessarily from the .hg/sparse file. The API should be updated later to pass in a filename so the error can be more descriptive. Copyright of the added file was copied from the sparse extension.

sparse: use vfs.tryread() vfs.exists() followed by a file read is an anti-pattern because it incurs an extra stat() to test for file presence. vfs.tryread() returns empty string on missing file and avoids the stat().

sparse: refactor sparsechecksum() This was relying on garbage collection to close the opened file, which is a bug. Both callers simply called into self.vfs to resolve the path. So refactor to use the vfs layer. While we're here, rename the method to reflect it is internal and to break anyone relying on the old behavior.

sparse: document config file format This was previously undocumented. Seems useful to have.

sparse: rename command to debugsparse Sparse checkout is still highly experimental and not protected by BC guarantees yet. We also haven't had a discussion on the UX. To discourage use, we rename the sparse command to debugsparse.

sparse: remove reference to simplecache This is a 3rd party extension authored by Facebook. References in core are not appropriate. It will be possible to restore this code/optimization via monkeypatching. So Facebook won't lose any functionality. The removed code is important for performance. So add a comment tracking it.

sparse: remove reference to hgwatchman This is a legacy extension. Now that the extension is in core, we only need to support what's in core, which is fsmonitor.

sparse: expand module docstring Clarify lack of BC guarantees. And say a bit more about the extension.

sparse: vendor Facebook-developed extension Facebook has developed an extension to enable "sparse" checkouts - a working directory with a subset of files. This feature is a critical component in enabling repositories to scale to infinite number of files while retaining reasonable performance. It's worth noting that sparse checkout is only one possible solution to this problem: another is virtual filesystems that realize files on first access. But given that virtual filesystems may not be accessible to all users, sparse checkout is necessary as a fallback. Per mailing list discussion at https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-March/095868.html we want to add sparse checkout to the Mercurial distribution via roughly the following mechanism: 1. Vendor extension as-is with minimal modifications (this patch) 2. Refactor extension so it is more clearly experimental and inline with Mercurial practices 3. Move code from extension into core where possible 4. Drop experimental labeling and/or move feature into core after sign-off from narrow clone feature owners This commit essentially copies the sparse extension and tests from revision 71e0a2aeca92a4078fe1b8c76e32c88ff1929737 of the https://bitbucket.org/facebook/hg-experimental repository. A list of modifications made as part of vendoring is as follows: * "EXPERIMENTAL" added to module docstring * Imports were changed to match Mercurial style conventions * "testedwith" value was updated to core Mercurial special value and comment boilerplate was inserted * A "clone_sparse" function was renamed to "clonesparse" to appease the style checker * Paths to the sparse extension in tests reflect built-in location * test-sparse-extensions.t was renamed to test-sparse-fsmonitor.t and references to "simplecache" were removed. The test always skips because it isn't trivial to run it given the way we currently run fsmonitor tests * A double empty line was removed from test-sparse-profiles.t There are aspects of the added code that are obviously not ideal. The goal is to make a minimal number of modifications as part of the vendoring to make it easier to track changes from the original implementation. Refactoring will occur in subsequent patches.

contrib: widen "direct use of `python`" net again I think I've now caught all of them. Differential Revision: https://phab.mercurial-scm.org/D15

tests: clean up a newly-introduced instance of `python` Differential Revision: https://phab.mercurial-scm.org/D16

tests: clean up even more direct `python` calls with $PYTHON This time ones that are prefixed with =, ", ', or `. This appears to be the last of them. Differential Revision: https://phab.mercurial-scm.org/D14

contrib: widen the "don't use `python`" net a little I'm still cleaning this up, but it's easier to do in bite-size chunks like this than all at once. The negative lookahead avoids one false positive category from some output related to finding Subversion bindings. Differential Revision: https://phab.mercurial-scm.org/D13

followlines: join merge parents line ranges in blockdescendants() (issue5595) In blockdescendants(), we had an assertion when line range of a merge changeset was not consistent depending on which parent was considered for computation. For instance, this might occur when file content (in lookup range) is significantly different between parent branches of the merge as demonstrated in added tests (where we almost completely rewrite the "baz" file while also introducing similarities with its content in the other branch we later merge to). Now, in such case, we combine line ranges from all parents by storing the envelope of both line ranges. This is conservative (the line range is extended, possibly unnecessarily) but at least this should avoid missing descendants with changes in a range that would fall in that of one parent but not in another one (the case of "baz: narrow change (2->2+)" changeset in tests).

workingfilectx: add exists, lexists Switch the lone call in merge.py to use it. As with past refactors, the goal is to make wctx hot-swappable with an in-memory context in the future. This change should be a no-op today.

vfs: add explanation about cost of checkambig=True in corner case

vfs: replace avoiding ambiguity in abstractvfs.rename with _avoidambig This centralizes common logic to forcibly avoid file stat ambiguity into _avoidambig(), which was introduced by previous patch.

vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing Now, files (to be truncated) are opened with checkambig=True, only if localrepository caches it. Therefore, straightforward "copy if EPERM" is always reasonable to avoid file stat ambiguity at closing. This patch makes checkambigatclosing close wrapper copy the target file, and advance mtime on it after renaming, if EPERM. This can avoid file stat ambiguity, even if the target file is owned by another (see issue5418 and issue5584 for detail). This patch factors main logic out instead of changing checkambigatclosing._checkambig() directly, in order to reuse it.

transaction: apply checkambig=True only on limited files for similarity Now, transaction can determine whether avoidance of file stat ambiguity is needed for each files, by blacklist "checkambigfiles". For similarity to truncation in _playback(), this patch apply checkambig=True only on limited files in code paths below. - restoring files by util.copyfile(), in _playback() (checkambigfiles itself is examined at first, because it as a keyword argument might be None) - writing files at finalization of transaction, in _generatefiles() This patch reduces cost of checking stat at writing out and restoring files, which aren't filecache-ed.

transaction: avoid file stat ambiguity only for files in blacklist Advancing mtime by os.utime() fails for EPERM, if the target file is owned by another. bff5ccbe5ead and related changes made some code paths give advancing mtime up in such case, to fix issue5418. This causes file stat ambiguity (again), if it is owned by another. https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan To avoid file stat ambiguity in such case, especially for .hg/dirstate, ed66ec39933f made vfs.rename() copy the target file, and advance mtime of renamed one again, if EPERM (see issue5584 for detail). But straightforward "copy if EPERM" isn't reasonable for truncation of append-only files at rollbacking, because rollbacking might cost much for truncation of many filelogs, even though filelogs aren't filecache-ed. Therefore, this patch introduces blacklist "checkambigfiles", and avoids file stat ambiguity only for files specified in this blacklist. This patch consists of two parts below, which should be applied at once in order to avoid regression. - specify 'checkambig=True' at vfs.open(mode='a') in _playback() according to checkambigfiles - invoke _playback() with checkambigfiles - add transaction.__init__() checkambigfiles argument, for _abort() - make localrepo instantiate transaction with _cachedfiles - add rollback() checkambigfiles argument, for "hg rollback/recover" - make localrepo invoke rollback() with _cachedfiles After this patch, straightforward "copy if EPERM" will be reasonable at closing the file opened with checkambig=True, because this policy is applied only on files, which are listed in blacklist "checkambigfiles".

localrepo: store path and vfs location of cached properties This information is used to make transaction handle these files specially, in order to avoid file stat ambiguity of them. Gathering information about cached files via annotation classes can avoid overlooking properties newly introduced in the future.

template: add successors template Add a 'successorssets' template that returns the list of all closest known sucessorssets for a changectx. The elements of the list are changesets. The "closest successors" are the first locally known revisions encountered while, walking successors markers. It uses successorsets previously modified to support the closest argument. This logic respect repository filtering. So hidden revision will be skipped by this logic unless --hidden is specified. Since we only display the visible predecessors, this template will not display anything in most case. It makes a good candidate for inclusion in the default log output. I updated the test-obsmarker-template.t test file introduced with the predecessors template to test successorssets template.

template: add tests for more complex cases We add new tests for improving the coverage of existing obs-markers related template (predecessors) and the new one we are introducing (successorssets).

obsolete: closest divergent support Add a closest argument to successorssets changing the definition of latest successors. With "closest=false" (current behavior), latest successors are "leafs" on the obsmarker graph. They don't have any successor and are known locally. With "closest=true", latest successors are the closest locally-known changesets that are visible in the repository or repoview. Closest successors can be then obsolete, orphan. This will be used in a later patch to show the closest successor of changesets with the successorssets template.

obsolete: pass cache argument of successors set explicitly We plan to add a new argument to successorsets. But first we need to update all callers to pass cache argument explicitly to avoid arguments confusion.

obsolete: small doc update for 'successorssets' Clarify successorssets documentation before we start updating the main function. This patch serie will introduce the successorssets template, the opposite of predecessor templates. Successors will use successorssets function and requires some improvement so before doing that, we clean up successorssets a bit.

phabricator: do not read a same revision twice It's possible to set up non-linear dependencies in Phabricator like: o D4 |\ | o D3 | | o | D2 |/ o D1 The old `phabread` code will print D1 twice. This patch adds de-duplication to prevent that. Test Plan: Construct the above dependencies in a Phabricator test instance and make sure the old code prints D1 twice while the new code won't.

patch: make parsepatch optionally trim context lines Previously there is a suspicious `if False and delta > 0` which dates back to the beginning of hgext/record.py (b2607267236d). The "trimming context lines" feature could be useful (and is used by the next patch). So let's enable it. This patch adds a new `maxcontext` parameter to `recordhunk` and `parsepatch`, changing the `if False` condition to respect it. The old `trimcontext` implementation is also wrong - it does not update `toline` correctly and it does not do the right thing for `before` context. A doctest was added to guard us from making a similar mistake again. Since `maxcontext` is set to `None` (unlimited), there is no behavior change.

phabricator: try to fetch differential revisions in batch Previously, we read Differential Revisions one by one by calling `differential.query`. Fetching them one by one is suboptimal. Unfortunately, there is no Conduit API that allows us to get a stack of diffids using a single API call. This patch tries to be smarter using a simple heuristic: when fetching D59 as a stack, previous IDs like D51, D52, D53, ..., D58 are likely belonging to a same stack so just fetch them as well. Since `differential.query` only returns cheap metadata without expensive diff content, it shouldn't be a big problem for the server. Using a test Phabricator instance, this patch reduces `phabread` reading a 10 patch stack from about 13 to 30 seconds to 8 seconds.

phabricator: avoid calling differential.getcommitmessage Previously, we call differential.getcommitmessage API to get commit messages. Now we read that from "Differential Revision" object fetched via "differential.query" API. This removes one API call per patch.

phabricator: rework phabread to reduce memory usage and round-trips This patch reworked phabread a bit so it fetches the lightweight "Differential Revision" metadata for a stack first. Then read other data. This allows the code to: a) send 1 request to get all `hg:meta` data instead of N requests b) patches are read in desired order, no need to buffer the output "b)" reduces the memory usage from O(N^2) to O(N) since we no longer keep old patch contents in memory. The above `N` means the number of patches in the stack.

phabricator: abort if phabsend gets empty revs Previously we didn't abort. Now we abort if revs is empty. This is consistent with "hg export" behavior. Maybe "return 1" is also a reasonable behavior, but that's inconsistent with the existing "hg export".

phabricator: do not upload new diff if nothing changes Previously, `phabsend` uploads new diffs as long as the commit hash changes. That's suboptimal because sometimes the diff is exactly the same as before, the commit hash change is caused by a parent hash change, or commit message change which do not affect diff content. This patch adds a check examining actual diff contents to skip uploading new diffs in that case.

phabricator: add node and p1 to hg:meta property The "hg:meta" property is for extra metadata to reconstruct the patch. Previously it does not have node or parent information since I think by reading the patch again, the commit message will be mangled (like, added the "Differential Revision" line) and we cannot preserve the commit hash. However, the "parent" information could be useful. It could be helpful to locate the "base revision" so in case of a conflict applying the patch directly, we might be able to use 3-way merge to resolve it correctly. Note: "local:commits" is an existing "property" used by Phabricator that has the node and parent information. However, it lacks of timezone information and requires "author" and "authorEmail" to be separated. So we are using a different "property" - "hg:meta" to be distinguished from "local:commits".

phabricator: check associated Differential Revision from commit message Previously, only tags can "associate" a changeset to a Differential Revision. But the usual pattern (arc patch or hg phabread) is to put the Differential Revision URL in commit message. This patch makes the code read commit message to find associated Differential Revision if associated tags are not found. This makes some workflows possible. For example, if the author loses their repo, or switch to another computer, they can continue download their own patches from Phabricator and update them without needing to manually create tags.

tests: replace yet more calls to `python` with $PYTHON These are some simple cases. More to come in a future change. Reviewers: krbullock Reviewed By: krbullock Differential Revision: https://phab.mercurial-scm.org/D4

tests: capitalize Python when it's not used as a command name This avoids some false positives in an upcoming check-code rule. Reviewers: krbullock Reviewed By: krbullock Differential Revision: https://phab.mercurial-scm.org/D3

merge with stable

vfs: drop the 'mustaudit' API There are no remaining users of 'mustaudit' so we can safely drop the API. External user are unlikely from a quick research so no deprecation is added.

streamclone: stop using 'vfs.mustaudit = False' Now that each call disable the auditing on its own, we can safely drop this the mustaudit usage. No other code is modified.

vfs: simplify path audit disabling in stream clone The whole 'mustaudit' API is quite complex compared to its actual usage by its unique user in stream clone. Instead we add a "auditpath" parameter to 'vfs.__call_'. The stream clone code then explicitly open files with path auditing disabled. The 'mustaudit' API will be cleaned up in the next changeset.

auditor: add simple comment about repo.auditor and al Every once in a while, I get confused by what these are. Let us add a comment.

tag: make sure the repository is locked when tagging Otherwise, writing localtag can happen without the lock.

test: glob a line number in test-devel-warnings.t This make is simpler to edit the extensions file without side effect.

test: add a small comment to explain a section of test-devel-warning This makes each test boundaries clearer.

obsolete: reports the number of local changeset obsoleted when unbundling This is a first basic visible usage of the changes tracking in the transaction. We adds a new function computing the pre-existing changesets obsoleted by a transaction and a transaction call back displaying this information. Example output: added 1 changesets with 1 changes to 1 files (+1 heads) 3 new obsolescence markers obsoleted 1 changesets The goal is to evolve the transaction summary into something bigger, gathering existing output there and adding new useful one. This patch is a good first step on this road. The new output is basic but give a user to the content of tr.changes['obsmarkers'] and give an idea of the new options we haves. I expect to revisit the message soon. The caller recording the transaction summary should also be moved into a more generic location but further refactoring is needed before it can happen.

transaction: track new obsmarkers in the 'changes' mapping The obsstore collaborate with transaction to make sure we track all the obsmarkers added during a transaction. This will be useful for various usages: hooks, caches, better output, etc. This is the seconds kind of data added to tr.changes (first one was added revisions)

configitems: register the 'progress.estimate' config

configitems: register the 'progress.clear-complete' config

configitems: register the 'progress.assume-tty' config

configitems: register the 'format.usestore' config