changegroup: consolidate tree manifests sending into cg1packer Previously, we overrode a method to control how manifests were serialized. This method was redefined on cg3packer to send tree manifests. This commit moves the tree manifests sending variation to cg1packer and teaches the cgpacker constructor to control which version to use. After these changes, cg3packer was empty. So it has been removed. Differential Revision: https://phab.mercurial-scm.org/D4081

changegroup: pass end of manifests marker into constructor cg3 inserts a custom marker in the stream once all manifests have been transferred. This is currently abstracted out by overriding a method. Let's pass the end of manifests marker in as an argument to avoid the extra method. Differential Revision: https://phab.mercurial-scm.org/D4080

changegroup: pass function to build delta header into constructor Previously, the delta header struct format was defined on each class and each class had a separate function for building the delta header. We replace both of these with an argument to __init__ containing a callable that can format a delta header given a revisiondelta instance. Differential Revision: https://phab.mercurial-scm.org/D4079

changegroup: make delta header struct formatters actual structs Why we weren't using compiled Struct instances, I don't know. They make code simpler. In theory they are faster. Although I don't believe I was able to measure any meaningful change. That could be because this code is often dominated by compression, deltafication, and function call overhead. Differential Revision: https://phab.mercurial-scm.org/D4078

changegroup: pass version into constructor Currently, the version is an attribute on each class. Passing the argument into the constructor gets us one step closer to eliminating cg2packer and cg3packer. Differential Revision: https://phab.mercurial-scm.org/D4077

changegroup: define functions for creating changegroup packers Currently, we have 3 classes for changegroup generation. Each class handles a specific changegroup format. And each subsequent version's class inherits from the previous one. The interface for the classes is not very well defined and a lot of version-specific behavior is behind overloaded functions. This approach adds complexity and makes changegroup generation difficult to reason about. Upcoming commits will be consolidating these 3 classes so differences between changegroup versions and changegroup generation are controlled by parameters to a single constructor / type rather than by overriding class attributes via inheritance. We begin this process by building dedicated functions for creating each changegroup packer instance. Currently they just call the constructor on the appropriate class. This will soon change. Differential Revision: https://phab.mercurial-scm.org/D4076

changegroup: capture revision delta in a data structure The current changegroup generation code is tightly coupled to the revlog API. This tight coupling makes it difficult to implement alternate storage backends without requiring a large surface area of the revlog API to be exposed. This is not desirable. In order to support changegroup generation with non-revlog storage, we'll need to abstract the concept of delta generation. This commit is the first step down that road. We introduce a data structure for representing a delta in a changegroup. The API still leaves a lot to be desired. But at least we now have separation between data and actions performed on it. As part of this, we tweak behavior slightly: we no longer concatenate the delta prefix with the metadata header. Instead, we track and emit the prefix as a separate chunk. This shouldn't have any meaningful impact since all the chunks just get sent to the wire, the compressor, etc. Because we're introducing a new object, this does add some overhead to changegroup execution. `hg perfchangegroupchangelog` on my clone of the Mercurial repo (~40,000 visible revisions in the changelog) slows down a bit: ! wall 1.268600 comb 1.270000 user 1.270000 sys 0.000000 (best of 8) ! wall 1.419479 comb 1.410000 user 1.410000 sys 0.000000 (best of 8) With for `hg bundle -t none-v2 -a /dev/null`: before: real 6.610 secs (user 6.460+0.000 sys 0.140+0.000) after: real 7.210 secs (user 7.060+0.000 sys 0.140+0.000) I plan to claw back this regression in future commits. And I may even do away with this data structure once the refactor is complete. For now, it makes things easier to comprehend. Differential Revision: https://phab.mercurial-scm.org/D4075

changegroup: inline ellipsisdata() There's only one caller of it. I don't think it needs to exist as a standalone function. Differential Revision: https://phab.mercurial-scm.org/D4074

changegroup: rename "revlog" variables "revlog" shadows the module import. But more importantly, changegroup generation should be storage agnostic and not assume the existence of revlogs. Let's rename the thing providing revision storage to "store" to reflect this ideal property. Differential Revision: https://phab.mercurial-scm.org/D4073

changegroup: move generate() modifications from narrow Narrow had a custom version of generate() that was essentially a copy of generate() with inline additions to facilitate ellipses serving. This commit inlines those modifications into generate(). Differential Revision: https://phab.mercurial-scm.org/D4067

changegroup: move generatefiles() from narrow The code is a bit ugly in that it overrides the linknodes function that is passed in as a function. I'd like to think that the caller of generatefiles() would pass in the appropriate function. We can clean this up later. Differential Revision: https://phab.mercurial-scm.org/D4066

changegroup: move _sortgroup() from narrow Differential Revision: https://phab.mercurial-scm.org/D4065

changegroup: move close() from narrow More of the same. Differential Revision: https://phab.mercurial-scm.org/D4064

changegroup: move revchunk() from narrow The monkeypatched revchunk for ellipses serving is a completely independent implementation. We model it as such in the changegroup code. revchunk() is now a simple proxy function. Again, I wish we had better APIs here. Especially since this narrow code is part of cg1packer and cg1packer can't be used with narrow. Class inheritance is wonky. And I will definitely be making changes to changegroup code for delta generation. As part of the code move, `node.nullrev` was replaced by `nullrev`. And a reference to `orig` was replaced to call `self._revchunknormal` directly. Differential Revision: https://phab.mercurial-scm.org/D4063

changegroup: move deltaparent() from narrow I'm not keen on performing the attribute sniff to test for presence of ellipses mode: I'd rather we use a separate packer instance that was ellipses mode specific. But I've tried to formalize a better API without narrow in core and I can't make sense of all the monkeypatching. My goal is to inline as much of the monkeypatching as possible then refactor the changegroup generation API. We add this code to the cg2packer because narrow doesn't work with cg1. Differential Revision: https://phab.mercurial-scm.org/D4062

changegroup: move _packellipsischangegroup() from narrow The behavior here is not ideal, as the function constructs a packer then adds attributes to it. This will be cleaned up in subsequent commits. Moving this code is necessary to move the remainder of the bundle2-level changegroup part generation code into core. Differential Revision: https://phab.mercurial-scm.org/D4061

changegroup: move ellipsisdata() from narrow This is a pretty straightforward copy of the function. Differential Revision: https://phab.mercurial-scm.org/D4060

fileset: narrow status computation by left-hand-side of 'and' node Timing with warm disk cache: $ hg status --cwd mozilla-central 'set:path:build/ and unknown()' --time (orig) time: real 1.970 secs (user 1.560+0.000 sys 0.410+0.000) (new) time: real 0.330 secs (user 0.310+0.000 sys 0.020+0.000)

fileset: move copy constructor of matchctx near __init__

fileset: build status according to 'withstatus' hint _switchcallers is no longer needed since 'withstatus' node is reinserted for arguments of functions like revs(). New matchctx instance is created per 'withstatus' to make sure that status tuple is available only for children of the 'withstatus' node.

fileset: insert hints where status should be computed This will allow us to compute status against a narrowed set of files. For example, "path:build/ & (unknown() + missing())" is rewritten as "path:build/ & <withstatus>(unknown() + missing(), 'unknown missing')", and the status call can be narrowed by the left-hand-side matcher, "path:build/". mctx.buildstatus() calls will be solely processed by getmatchwithstatus().

fileset: move buildstatus() to matchctx method In future patches, file status will be computed while evaluating a parsed tree. This patch provides a matchctx interface to build status.

fileset: keep basectx by matchctx

fileset: pass in basectx to _buildstatus() I'll make matchctx remember both ctx and basectx so that file status between them can be computed later. This prepares for the change.

resolve: update commands.resolve.confirm help text Included --mark and --unmark in the help text of resolve.confirm.config. Differential Revision: https://phab.mercurial-scm.org/D4103

resolve: support confirm config option with --unmark flag Now, commands.resolve.confirm also respect --unmark option; and confirm to unresolve all resolved files. It will confirm only when no files pats are passed (same as --mark), because when no pats are passed the default is to mark resolved files as unresolved. And if user has passed file pats then I think there is no need to confirm for that. Differential Revision: https://phab.mercurial-scm.org/D4102

resolve: correct behavior of mark-check=none to match docs Differential Revision: https://phab.mercurial-scm.org/D4121

narrow: move .hg/narrowspec to .hg/store/narrowspec (BC) The narrowspec is more closely related to the store than to the working copy. For example, if the narrowspec changes, the set of revlogs also needs to change (the working copy may change, but that depends on which commit is checked out). Also, when using the share extension, the narrowspec needs to be shared along with the store. This patch therefore moves the narrowspec into the store/ directory. This is clearly a breaking change, but I haven't bothered trying to fall back to reading the narrowspec from the old location (.hg/), because there are very few users of narrow out there. (We'll add a temporary hack to our Google-internal extension to handle the migration.) Differential Revision: https://phab.mercurial-scm.org/D4099

narrow: drop checkambig=True when restoring backup IIUC, checkambig is about updating timestamps of the file while renaming. That's important for the dirstate, but we never check the timestamp of the narrowspec file. We can therefore avoid checking passing checkambig=True. Differential Revision: https://phab.mercurial-scm.org/D4098

narrow: remove a repo file-cache invalidation It's unclear why this was needed. All tests pass without it. I asked Kyle Lippincott (who added the check) and he also doesn't remember what it was for. Differential Revision: https://phab.mercurial-scm.org/D4097

narrow: call narrowspec.{save,restore,clear}backup directly I want to move .hg/narrowspec to .hg/store/narrowspec and we need to decouple the narrowspec update from the dirstate update for that. This patch lets the callers call the narrowspec backup functions directly, in addition to the dirstate backup functions. The narrowspec methods are made to check if narrowing is enabled. For that, a repo instance was needed, which all the callers luckily already had available. Differential Revision: https://phab.mercurial-scm.org/D4096

index: don't add 1 to length variables A lot of "+ 1" and "-1" were mechanically added to ease the transition in 781b2720d2ac (index: don't include nullid in len(), 2018-07-20). Let's clean it up now. Differential Revision: https://phab.mercurial-scm.org/D4106

index: drop support for nullid at position len(index) in index_node I think no callers exist since at least a3dacabd476b (index: don't allow index[len(index)] to mean nullid, 2018-07-20). Differential Revision: https://phab.mercurial-scm.org/D4105

index: return False for "len(index) in index" Since we no longer accept index[len(index)], we should clearly make "len(index) in index" return False. This should have been part of a3dacabd476b (index: don't allow index[len(index)] to mean nullid, 2018-07-20) Differential Revision: https://phab.mercurial-scm.org/D4104

fileset: combine union of basic patterns into single matcher This appears to improve query performance in a big repository than I thought. Writing less Python in a hot loop, faster computation we gain. $ hg files --cwd mozilla-central --time 'set:a* + b* + c* + d* + e*' (orig) time: real 0.670 secs (user 0.640+0.000 sys 0.030+0.000) (new) time: real 0.210 secs (user 0.180+0.000 sys 0.020+0.000)

fileset: reorder 'or' expression by weight

fileset: introduce weight constants for readability These constants are defined in the filesetlang module since it's the bottommost module depending on WEIGHT_CHECK_FILENAME, and extensions will be likely to import it to process function arguments. Credit for the naming goes to Augie Fackler.

sparse: use named parameters in i18n strings This should give more hints about what the %s means, and allow reordering.

sparse: do not include operator in i18n strings

fix: compute changed lines lazily to make whole-file fixer tools faster Differential Revision: https://phab.mercurial-scm.org/D4100

resolve: support commands.resolve.confirm option with --mark flag Now, commands.resolve.confirm config option also respect --mark; and confirm to mark all 'unresolved' files as 'resolved'. It will confirm only when you don't pass any pats i.e 'hg resolve -m', because when no file pats are passed then --mark's default functionality is to mark all unresolved files. And if user has given file pats then I think there is no need to confirm. Differential Revision: https://phab.mercurial-scm.org/D4101

resolve: add commands.resolve.mark-check=abort to tweakdefaults Differential Revision: https://phab.mercurial-scm.org/D4072

resolve: graduate resolve.mark-check from experimental, add docs Since this hasn't been in a release yet, I'm not bothering to add an alias for the experimental name of the config variable. Differential Revision: https://phab.mercurial-scm.org/D4071

shortest: use 'x' prefix to disambiguate from revnum if configured Differential Revision: https://phab.mercurial-scm.org/D4042

revisions: allow "x123" to refer to nodeid prefix "123" When resolving "123" to a revision, we try to interpret it as revnum before we try to interpret it as a nodeid hex prefix. This can lead to the shortest valid prefix being longer than necessary. This patch lets us write such nodeids in a shorter form by prefixing them with "x" instead of adding more hex digits until they're longer than the longest decimal revnum. On my hg repo with almost 69k revisions, turning this feature on saves on average 0.4% on the average nodeid length. That clearly doesn't justify this patch. However, it becomes more usefule when combined with the earlier patches in this series that let you disambiguate nodeid prefixes within a configured revset. Note that we attempt to resolve symbols as nodeid prefixes after we've exhausted all other posibilities, so this is a backwards compatible change (only queries that would previously fail may now succeed). I've still hidden this feature behind an experiemntal config option so we can roll it back if needed. Differential Revision: https://phab.mercurial-scm.org/D4041

shortest: make isrev() a top-level function I'm going to add another caller in the next patch. Differential Revision: https://phab.mercurial-scm.org/D4040

shortest: cache disambiguation revset This makes it actually useful. In compared the time in my hg repo with 69.6k revisions and with a disambiguation revset of "not public()" that matches 563 visible revisions. I ran "time hg log -T '{shortest(node1,)}' -r 0:1000" (no revisions within the revset in that revision range). Before this patch, it took 57s and after it took 0.7s. Differential Revision: https://phab.mercurial-scm.org/D4039

index: don't allow index[len(index)] to mean nullid Now everything else has been cleaned up and we can drop support for getting the nullid from the end of the index (from *past* the end actually, since we reduced the length in the previous patch). Differential Revision: https://phab.mercurial-scm.org/D4023

index: don't include nullid in len() I suspect the reason the nullid is in the index in the last position is that it lets index[i] for regular revision number, even when index was just a regular Python list. An alternative solution would have been to reserve revision number 0 for the null revision. I don't know why that wasn't done. Now that we have classes backing the index, we can easily make index[-1] get the nullid without having to put it last in the list and including it in the len(). This patch just hides the nullid -- it will still be accessible at index[len(index)]. I realize that this will be annoying when checking out across this commit for debugging (including bisection). Differential Revision: https://phab.mercurial-scm.org/D4022

index: replace insert(-1, e) method by append(e) method I want to make index[len(index) - 1] be the tip revision, not null revision as it is today. insert(-1, e) will then make little sense. Since insert() currently requires the first argument to be -1, it seems simpler to replace it by a method that allows insertion only at the end. Note that revlogoldindex already has this method (by virtue of extending list). Differential Revision: https://phab.mercurial-scm.org/D4021

pure: create type for revlog v0 index The new type takes care of hiding the nullid entry. This type helps us transition away from including that entry at index[len(index)-1]. Differential Revision: https://phab.mercurial-scm.org/D4020

index: drop support for negative indexes into the index I want index[i] to work for any valid revnum including -1 (as it already does), and I also want len(index) to be the number of revisions in the index (not counting the null revision), so it cannot also support negative revision numbers other than -1 for nullid. I didn't bother removing support for it from revlog v0. Differential Revision: https://phab.mercurial-scm.org/D4019

index: handle index[-1] as nullid more explicitly I find it more intuitive to check if "pos == -1" than to first add the index length (which includes one extra item for the nullid) and compare that to "length - 1". However, because test-parseindex2.py compares the whole index (up to len(index)-1), we need to also preserve that other check for a little while more. I'll remove it soon. Differential Revision: https://phab.mercurial-scm.org/D4018

index: store nullrev as -1 in nodetree Nothing important, it just seems more natural to not map nullrev to INT_MAX. We just need to change the revision encoding a little to make space for the -1. Differential Revision: https://phab.mercurial-scm.org/D4005

index: create function for deleting node from nodetree A node is marked deleted by nt_insert(self, node, -1), but "-1" could mean so many things (notably nullrev), so let's hide it in a function. The function could possibly also remove nodes from the tree in the future (it currently only marks them deleted). Differential Revision: https://phab.mercurial-scm.org/D4004

revlog: remove some knowledge of sentinel nullid in index I think the "-2" to mean "last position in index, not counting the null revision at the end" is an implementation detail of the index that we should avoid spreading knowledge of. I hope we can even remove support for index[-2]. Differential Revision: https://phab.mercurial-scm.org/D4016

scmutil: make shortest() respect disambiguation revset The previous patch would let you use a shorter prefix if the prefix is unique within a configured revset. However, that's not very useful if there's no simple way of knowing what that shorter prefix is. This patch adapts the shortest() template function to use the shorter prefixes for nodes in the configured revset. This is currently extremely slow, because it calculates the revset for each call to shortest(). To make this faster, the next patch will start caching the revset instance. Ideally we'd cache a prefix tree instance instead. Differential Revision: https://phab.mercurial-scm.org/D4038

lookup: add option to disambiguate prefix within revset When resolving a nodeid prefix that is not unique within the repo and the user has configured a revset that they want to disambiguate within, we now try to look up within that revset before we fail. If there is a unique match within the revset, we use that. This is of course most effective at allowing a short prefix if the revset contains few nodes. For most of our internal users at Google, "not public()" is sufficiently small that a hex digit or two is enough. The implementation is currently pretty slow, but good enough for small revsets (which is the expected use case). The scan in the revset is linear. We may want to use a prefix tree if we want to allow users to use a larger revset. Credit for the idea goes to Kyle Lippincott. Differential Revision: https://phab.mercurial-scm.org/D4037

revlog: use specialized exception for ambiguous prefix lookup It's useful to be able to catch a specific exception for this case. We'll use it soon. Differential Revision: https://phab.mercurial-scm.org/D4036

narrowspec: remove the unused _parsestoredpatterns() function Previous patch started using sparse.parseconfig() to read narrowspec and now we don't need this function. Differential Revision: https://phab.mercurial-scm.org/D4058

narrowspec: use sparse.parseconfig() to parse narrowspec file (BC) This also make narrow files use 'include' and 'exclude' instead of plural forms which are 'includes' and 'excludes'. This is BC because existing narrowspecs have to replace excludes, includes words with exclude and include i.e. their singular versions. Differential Revision: https://phab.mercurial-scm.org/D4057

sparse: add an action argument to parseconfig() This will help us in reusing this function to parse narrow config files and unfiying the config file parsing logic. Differential Revision: https://phab.mercurial-scm.org/D4056

tests: avoid unnecessarily looking inside .hg/ in test-narrow* Many of the narrow tests were looking at .hg/narrowspec. I think most of them were written before we added `hg tracked`, but now that we have that command, we should fix the tests. There were also a few instances of tests looking at .hg/dirstate. I fixed those to use `hg files` instead. Differential Revision: https://phab.mercurial-scm.org/D4093

narrow: extract part of narrowspec backup to core Just a little preparation for the next patch. Differential Revision: https://phab.mercurial-scm.org/D4095

narrow: move requirement constant from changegroup to repository As suggested by Gregory Szorc. Differential Revision: https://phab.mercurial-scm.org/D4094

merge with stable

tests: update test expectations in pre-2.7.9 branch of this test As far as I can tell this is the only spot that got overlooked. Rough test methodology (run inside an hg working copy): docker run --rm -v `pwd`:/hg -it debian:wheezy bash apt-get update apt-get install python python-dev build-essential unzip mercurial \ wget libbz2-dev make testpy-2.7.8 You could try and use the system Python, but it's 2.7.3 and has lots of issues for hg these days that are not worth fixing. Differential Revision: https://phab.mercurial-scm.org/D4070

fileset: optimize 'x and not y' to 'x - y' 'x - y' is first rewritten to 'x and not y' so that x and y are reordered by weight.

fileset: reorder 'and' expression to evaluate basic patterns first Timing of a crafted example (when disk cache is warm): $ hg files set:'binary() and path:contrib' (orig) time: real 0.140 secs (user 0.120+0.000 sys 0.020+0.000) (new) time: real 0.040 secs (user 0.030+0.000 sys 0.010+0.000)

fileset: roughly adjust weights of functions ... in order to move status predicates far away from basic patterns. I don't know if each weight is appropriate, but it should be good enough as a start.

fileset: add stub for weight-based optimization The main purpose of this change is to group basic patterns, which can be mapped to a plain matcher. I'm not so interested in a weight of each function.

fileset: reject 'negate' node early while transforming parsed tree That's how a 'negate' node is processed in revset.

fileset: drop 'group' node from tree to be evaluated This helps rewriting a parsed tree.

fileset: add phase to transform parsed tree This isn't strictly necessary, but I decided to just follow the strategy of the revset parsing.

py3: whitelist another 3 tests caught by the ratchet Differential Revision: https://phab.mercurial-scm.org/D4069

testrunner: allow multiple #testcases This lets you have multiple #testcases declarations and they're taken to be different dimensions of the test, so their cross product becomes the total set of test cases. For example: #testcases obsstore-on obsstore-off #testcases manifest-flat manifest-tree $ hg rebase ... ... #if obsstore-on $ hg log ... #endif Note that this is an excellent way to slow down the test suite, so use it with care. The feature is implemented by replacing most of the "case" variables that were strings before by an array of strings with each item a different dimension of the test case. The file names are created by joining the dimensions by "#" (e.g. test-foo.t#obsstore-on#manifest-flat). Differential Revision: https://phab.mercurial-scm.org/D4049

testrunner: use "#" for "test cases" suffix in .err filename too This seems like a natural follow-up to b865bba56db1 (run-tests: update the test case name format, 2018-05-13). Differential Revision: https://phab.mercurial-scm.org/D4052

resolve: add confirm config option This config setting gives a functionality to confirm before it re-merge all unresolved files. If this config is enabled, when you run 'hg resolve --all' it will prompt with a msg "re-merge all unresolved files (yn)?" To enable this functionality: [commands] resolve.confirm = True Differential Revision: https://phab.mercurial-scm.org/D3988

revlog: remove side effect from failed nt_init() If nt_init() successfully allocates memory for the node tree but then runs out of memory while trying to insert nullid into it, it will leave the node tree pointer set on the index object. That means that future node tree operations will think that the tree is properly initialized. It seems very unlikely to make a difference in practice (if nt_init() runs out of memory, not much else will probably work), but since I spotted it, I figured I might as well fix it. Differential Revision: https://phab.mercurial-scm.org/D4028

revlog: remove micro-optimization for looking up only nullid index_find_node() would call nt_find() before initializing the node tree. nt_find() would then return -2 unless the requested revision was the null revision. I can't imagine what scenario that is optimizing for, and doing the initialization earlier makes the code simpler and easier to follow, so that's what this patch does. Differential Revision: https://phab.mercurial-scm.org/D4027

revlog: remove unnecessary output parameter from node_check() The "nodelen" output parameter is always set to 20 if the function returns successfully. Differential Revision: https://phab.mercurial-scm.org/D4026

narrow: move status-filtering to core and to ctx One of my recent changes from repo.status(ctx1, ctx2) to ctx1.status(ctx2) broke some of our Google-internal tests. The problem turned out to be that the narrow extension was overriding repo.status() to make it filter out paths outside the narrowspec. When I changed to ctx1.status(ctx2), then that filtering obviously got lost. ctx.status() seems like a better method to do the filtering in, so this patch moves the filtering into that method, thereby also moving it out of the extension and into core. Differential Revision: https://phab.mercurial-scm.org/D4068

amend: support "history-editing-backup" config option Now, amend is in the list of those history editing commands which support `history-editing-backup` config option. If you don't want to store any backup then just use this config. [ui] hisotry-editing-backup = False Current status of list of history editing commands which support this config: 1. histedit 2. rebase 3. amend Differential Revision: https://phab.mercurial-scm.org/D3968

rebase: move "backup" flag to rebaseruntime It was getting messy to populate "backup" flag to calls of `_finishrebase()` and `_prepareabortorcontinue`, so made some changes to move "backup" flag to rbsrt. Differential Revision: https://phab.mercurial-scm.org/D4055

i18n: format warning of hggettext in standard compiler error style Now, hggettext specific warning messages are formatted in: FILENAME:LINENO:MESSAGE This allows editors to jump into corresponded line easily.

i18n: omit redundant translatable synopsis text to avoid xgettext warning This empty translatable synopsis text causes xgettext warning below: Empty msgid. It is reserved by GNU gettext: gettext("") returns the header entry with meta information, not the empty string. This "synopsis" argument of @command annotation is omitable.

i18n: avoid substitution of PYFILES at runtime for readability of output This substitution decreases readability of "make update-pot" output, because PYFILES consists of many files. This patch makes "make update-pot" show "find mercurial hgext doc -name '*.py'" instead of many *.py files at runtime.

util: make new timedcmstats class Python 3 compatible

fix: add a monkey-patchable point after all new revisions have been committed Differential Revision: https://phab.mercurial-scm.org/D4048

help: add quotes to a few commands we point to I didn't know that 'hg help "revsets.x or y"' was valid syntax, so the quoting is extra useful here to make it clear that that is an actual command. Differential Revision: https://phab.mercurial-scm.org/D4059

linelog: port to Python 3 Differential Revision: https://phab.mercurial-scm.org/D4051

exchange: move simple narrow changegroup generation from extension The narrow extension completely replaced the function generating the changegroup part when a narrow changegroup was requested. Previous commits have taught the in-core changegroup code how to filter files based on a matcher. This commit teaches the in-core bundle2 part generation code to construct a matcher based on arguments. It will also emit a bundle2 part describing the narrow spec. I believe the changegroup part generation code in the narrow extension is now limited to ellipsis serving mode. i.e. core is now capable of narrow changegroup generation when ellipsis mode is disabled. Differential Revision: https://phab.mercurial-scm.org/D4014

exchange: make narrow ACL presence imply narrow=True And refactor the logic for determining when to invoke our custom changegroup part function so it is only conditional on narrow being set. This makes it more obvious under which conditions we should invoke the custom implementation. Also use raw strings so **kwargs works on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4013

changegroup: inline prune() logic from narrow prune() needs to ellide manifests that aren't part of the narrow matcher. The code is violating storage abstractions, so a comment has been added. Keep in mind the impetus for moving this code to core is so changegroup code can be refactored to be storage agnostic. Differential Revision: https://phab.mercurial-scm.org/D4012

fileset: extract language processing part to new module (API) I'll add a couple more functions that work on parsed tree. % wc -l mercurial/fileset*.py 559 mercurial/fileset.py 135 mercurial/filesetlang.py 694 total

fileset: flatten 'or' nodes to unnest unionmatchers This also makes it easier to compile a union of basic patterns into a single regexp pattern.

fileset: flatten arguments list Just prepares for flattening 'or' nodes. This change would have no impact on performance.

debugfileset: add option to show matcher representation

debugfileset: backport --show-stage option from debugrevspec I'll add some static optimizations to fileset.

lfs: remove callstatus property from 'lfs()' fileset It was added at 91aac8e6604d, but is no longer needed since a fileset expression is now compiled into an "open" matcher. See ff5b6fca1082 for details.

rebase: support "history-editing-backup" config option If you don't want to store any backup while rebasing, you can use `history-editing-backup` config option. [ui] history-editing-backup = # True or False Current status of list of commands which supports this config: 1. histedit 2. rebase Differential Revision: https://phab.mercurial-scm.org/D3887

extensions: add detailed loading information This lets you track down what exactly is happening during extension loading, and how long various steps took.

util: create a context manager to handle timing The context manager is pulled out of the timed decorator function, and refactored to provide a stats instance, with added tests.

linelog: fix import statements to pass the import checker on py3 Differential Revision: https://phab.mercurial-scm.org/D4050

linelog: add a Python implementation of the linelog datastructure This datastructure was originally developed by Jun Wu at Facebook, inspired by SCCS weaves. It's useful as a cache for blame information, but also is the magic that makes `hg absorb` easy to implement. In service of importing the code to Mercurial, I wanted to actually /understand/ it, and once I did I decided to take a run at implementing it. The help/internals/linelog.txt document is the README from Jun Wu's implementaiton. It all applies to our linelog implementation. Differential Revision: https://phab.mercurial-scm.org/D3990

changegroup: move file matcher from narrow extension Sparse changegroup generation requires the use of a matcher to filter which files are relevant. This commit moves the file matcher from the narrow extension to core and updates the narrow extension to use it. I'm not sure why the narrow extension was storing the matcher as a callable that resolved to a matcher. So I changed it to be a simple matcher instance. In addition, code from narrow to intersect the matcher with the local narrow spec is now performed automatically when the changegroup packer is created. If a matcher is not passed into getbundler() an alwaysmatcher() is assumed. This ensures that a matcher is always defined for all operations. Differential Revision: https://phab.mercurial-scm.org/D4011

resolve: add option to warn/abort on -m with unresolved conflict markers When a user is dropped out of Mercurial to a terminal to resolve files, we emit messages like: conflicts while merging file1! (edit, then use 'hg resolve --mark') conflicts while merging file2! (edit, then use 'hg resolve --mark') We don't mention a file name in the hint, so some users might do something like `$EDITOR file1; hg resolve --mark`, see that it says "(no more unresolved files)" and forget to deal with file2 before running the next command. Even if we did mention a file name in the hint, it's too easy to forget it (maybe the merge spans a couple days or something). This option lets us inform the user that they might have missed something. In the scenario above, the output would be something like: warning: the following files still have conflict markers: file2 (no more unresolved files) Differential Revision: https://phab.mercurial-scm.org/D4035

exchange: refactor control flow of _getbundlechangegrouppart() The use of early return makes the control flow of this function much easier to reason about IMO. Differential Revision: https://phab.mercurial-scm.org/D4010

exchange: move _computeellipsis() from narrow This is also referenced as part of the narrow changegroup code and therefore needs to move to core before we can integrate the narrow changegroup code into core. Differential Revision: https://phab.mercurial-scm.org/D4009

exchange: move narrow acl functionality into core This function is called by the custom changegroup generation code in the narrow extension. I want to move that changegroup code into core. That means we need to move this function. The code is kinda hacky in that assumes existence of REMOTE_USER, which is only present on authenticated HTTP requests. I've added a comment indicating that. Differential Revision: https://phab.mercurial-scm.org/D4008

exchange: move disabling of rev-branch-cache bundle part out of narrow I'm attempting to refactor changegroup code in order to better support alternate storage backends. The narrow extension is performing a lot of monkeypatching to this code and it is making it difficult to reason about how everything works. I'm reasonably certain I would be unable to abstract storage without requiring extensive rework of narrow. I believe it is less effort to move narrow code into core so it can be accounted for when changegroup code is refactored. So I'll be doing that. The first part of this is integrating the disabling of the cache:rev-branch-cache bundle2 part into core. This doesn't seem like it is related to changegroup, but narrow's modifications to changegroup are invasive and also require taking its code for bundle generation and exchange into core in order for the changegroup code to work. Differential Revision: https://phab.mercurial-scm.org/D4007

dispatch: show a short error message when invalid global option given Similar reasoning as the previous patch. Differential Revision: https://phab.mercurial-scm.org/D4025

dispatch: don't show list of commands on bogus command If a command is ambiguous, you get this: $ hg ve hg: command 've' is ambiguous: verify version [255] If you typo a command, you get this: $ hg comit hg: unknown command 'comit' (did you mean one of commit, incoming, mycommit?) [255] But if you completely mistype a command so it no longer looks like any existing commands, you get a full list of commands. That might be useful the first time you use Mercurial, but after that it's probably more annoying than help, especially if you have the pager enabled and have a short terminal. Let's instead give a short hint telling the user to run `hg help` for more help. Differential Revision: https://phab.mercurial-scm.org/D4024

histedit: avoid repeating name of state file in a few places We can depend on the state object instead. Differential Revision: https://phab.mercurial-scm.org/D4006

py3: stop rewriting xrange() to pycompat.xrange() We now require the use of pycompat.xrange() in source. One less feature in the module importer gets us one step closer to Python 3 native source code. Differential Revision: https://phab.mercurial-scm.org/D4034

check-code: ban use of bare xrange() We want everyone to use pycompat.xrange(). Differential Revision: https://phab.mercurial-scm.org/D4033

global: use pycompat.xrange() On Python 3, our module importer automatically rewrites xrange() to pycompat.xrange(). We want to move away from the custom importer on Python 3. This commit converts all instances of xrange() to use pycompat.xrange(). Differential Revision: https://phab.mercurial-scm.org/D4032

pycompat: add xrange alias for Python 2 We assign xrange on Python 3 but not Python 2. Adding the missing symbol on Python 2 allows us to use `pycompat.xrange` to get a generator range function. Differential Revision: https://phab.mercurial-scm.org/D4031

manifest: persist the manifestfulltext cache Reconstructing the manifest from the revlog takes time, so much so that there already is a LRU cache to avoid having to load a manifest multiple times. This patch persists that LRU cache in the .hg/cache directory, so we can re-use this cache across hg commands. Commit benchmark (run on Macos 10.13 on a 2017-model Macbook Pro with Core i7 2.9GHz and flash drive), testing without and with patch run 5 times, baseline is r2a227782e754: * committing to an existing file, against the mozilla-central repository. Baseline real time average 1.9692, with patch 1.3786. A new debugcommand "hg debugmanifestfulltextcache" lets you inspect the cache, clear it, or add specific manifest nodeids to it. When calling repo.updatecaches(), the manifest(s) for the working copy parents are added to the cache. The hg perfmanifest command has an additional --clear-disk switch to clear this cache when testing manifest loading performance. Using this command to test performance on the firefox repository for revision f947d902ed91, whose manifest has a delta chain length of 60540, we see: $ hg perfmanifest f947d902ed91 --clear-disk ! wall 0.972253 comb 0.970000 user 0.850000 sys 0.120000 (best of 10) $ hg debugmanifestfulltextcache -a `hg log --debug -r f947d902ed91 | grep manifest | cut -d: -f3` Cache contains 1 manifest entries, in order of most to least recent: id: 0294517df4aad07c70701db43bc7ff24c3ce7dbc, size 25.6 MB Total cache data size 25.6 MB, on-disk 0 bytes $ hg perfmanifest f947d902ed91 ! wall 0.036748 comb 0.040000 user 0.020000 sys 0.020000 (best of 100) Worst-case scenario: a manifest text loaded from a single delta; in the firefox repository manifest node 9a1246ff762e is the chain base for the manifest attached to revision f947d902ed91. Loading this from a full cache file is just as fast as without the cache; the extra node ids ensure a big full cache: $ for node in 9a1246ff762e 1a1922c14a3e 54a31d11a36a 0294517df4aa; do > hgd debugmanifestfulltextcache -a $node > /dev/null > done $ hgd perfmanifest -m 9a1246ff762e ! wall 0.077513 comb 0.080000 user 0.030000 sys 0.050000 (best of 100) $ hgd perfmanifest -m 9a1246ff762e --clear-disk ! wall 0.078547 comb 0.080000 user 0.070000 sys 0.010000 (best of 100)

debug: allow specifying a manifest node rather than a revision