log: make changesetformatter pass in changectx to formatter It wasn't necessary before, but user templates may have keywords that aren't filled in by the changesetformatter.

journal: use changesetformatter to properly nest list of commits in JSON Before, two separate JSON documents were interleaved. I chose the field name "changesets" over the option name "commits", since each entry is called a "changeset" in log templates.

journal: do not pass in repolookuperror string to template (BC) This doesn't look like data, but a warning message.

journal: inline formatted nodes and date into expression The variable name "str" was misleading since these values aren't always strings.

journal: unify template name for "nodes" (BC) This is a part of the name unification. https://www.mercurial-scm.org/wiki/GenericTemplatingPlan#Dictionary .. bc:: ``{oldhashes}`` and ``{newhashes}`` in journal template are renamed to ``{oldnodes}`` and ``{newnodes}`` respectively.

localrepo: extract resolving of opener options to standalone functions Requirements and config options are converted into a dict which is available to the store vfs to consult. This is how storage options are communicated from the repo layer to the storage layer. Currently, we do that option resolution in a private method on the repo instance. And there is a single method doing that resolution. Opener options are logically specific to the storage backend they apply to. And, opener options may wish to influence how the repo object/type is constructed. So it makes sense to have more granular storage option resolution that occurs before the repo object is instantiated. This commit extracts the code for resolving opener options into new module-level functions. These functions are run before the repo instance is constructed. As part of the code move, we split the option resolution into generic and revlog-specific options. After this commit, we no longer add revlog-specific options to repos that don't have a revlog requirement. Some of these opener options and associated config options might make sense on alternate storage backends. We can always reuse config options and opener option names for other backends. But we shouldn't be passing opener options to storage backends that won't recognize them. I haven't done it here, but after this commit it should be possible for store backends to validate the set of opener options it receives. Because localrepository.openerreqs is no longer used after this commit, it has been removed. I'm not super thrilled about the code outside of localrepo that is adding requirements and updating opener options. We'll probably want to create a more formal API for that use case that constructs a new repo instance and poisons the old repo object. But this was a pre-existing issue and can be dealt with later. I have little doubt it will cause me troubles as I continue to refactor how repository objects are instantiated. .. api:: ``localrepository.openerreqs`` has been removed. Override ``localrepo.resolvestorevfsoptions()`` to add custom opener options. .. api:: ``localrepository._applyopenerreqs()`` has been removed. Use ``localrepo.resolvestorevfsoptions()`` to add custom opener options. Differential Revision: https://phab.mercurial-scm.org/D4576

localrepo: use boolean in opener options Not sure why we're using an integer for a flag value here. I'm pretty sure nothing relies on values being 1. While we're here, convert to a dict comprehension. Differential Revision: https://phab.mercurial-scm.org/D4575

localrepo: move store() from store module I want logic related to requirements handling to be in the localrepo module so it is all in one place. I would have loved to inline this logic. Unfortunately, statichttprepo also calls it. I didn't want to inline it twice. We could potentially refactor statichttppeer. But meh. Differential Revision: https://phab.mercurial-scm.org/D4574

localrepo: resolve store and cachevfs in makelocalrepository() This is mostly a code move and refactor. One change is that we now explicitly look for requirements indicating a share is being used rather than blindly try to read from .hg/sharedpath. Requirements *should* be all that is necessary to dictate high-level behavior and I'm not sure why the previous code was doing what it was. The previous code has been in place since 87d1fd40f57e (authored in 2009). And the commit immediately after that (971e38a9344b) introduced ``hg.share()`` and always wrote the ``shared`` requirement. And as far as I can tell, every revision of ``hg.share()`` since has written either the ``shared`` or ``relshared`` requirement. So I'm pretty sure we don't need to maintain BC by always looking for and honoring the ``.hg/sharedpath`` file even if a requirement isn't present. .. bc:: A repository will no longer use shared storage if it has a ``.hg/sharedpath`` file but no entry in ``.hg/requires`` saying it is shared. This change should not have any end-user impact, as all shared repos should have a ``.hg/requires`` file indicating this. Differential Revision: https://phab.mercurial-scm.org/D4573

localrepo: document and test bug around opening shared repos As part of refactoring this code, I realized that we don't validate the requirements of a shared repository. This commit documents that next to the requirements validation code and adds a test demonstrating the buggy behavior. I'm not sure if I'll fix this. But it is definitely a bug that users could encounter, as LFS, narrow, and potentially other extensions dynamically add requirements on first use. One part of this I'm not sure about is how to handle loading the .hg/hgrc of the shared repo. We need to do that in order to load extensions. But we don't want that repo's hgrc to overwrite the current repo's. Differential Revision: https://phab.mercurial-scm.org/D4572

localrepo: move requirements reasonability testing to own function Just because we know how to handle each listed requirement doesn't mean that set of requirements is reasonable. This commit introduces an extension-wrappable function to validate that a set of requirements makes sense. We could combine this with ensurerequirementsrecognized(). But I think having a line between basic membership testing and compatibility checking is more powerful as it will help differentiate between missing support and buggy behavior. Differential Revision: https://phab.mercurial-scm.org/D4571

statichttprepo: use new functions for requirements validation The new code in localrepo for requirements gathering and validation is more robust than scmutil.readrequires(). Let's port statichttprepo to it. Since scmutil.readrequires() is no longer used, it has been removed. It is possible extensions were monkeypatching this to supplement the set of supported requirements. But the proper way to do that is to register a featuresetupfuncs. I'm comfortable forcing the API break because featuresetupfuncs is more robust and has been supported for a while. .. api:: ``scmutil.readrequires()`` has been removed. Use ``localrepo.featuresetupfuncs`` to register new repository requirements. Use ``localrepo.ensurerequirementsrecognized()`` to validate them. Differential Revision: https://phab.mercurial-scm.org/D4570

localrepo: validate supported requirements in makelocalrepository() This should be a glorified code move. I did take the opportunity to refactor things. We now have a separate function for gathering requirements and one for validating them. I also mode cosmetic changes to the code, such as not using abbreviations and using a set instead of list to model missing requirements. Differential Revision: https://phab.mercurial-scm.org/D4569

localrepo: read requirements file in makelocalrepository() Previously, scmutil.readrequires() loaded the requirements file and validated its content against what was supported. Requirements translate to repository features and are critical to our plans to dynamically create local repository types. So, we must load them in makelocalrepository() before a repository instance is constructed. This commit moves the reading of the .hg/requires file to makelocalrepository(). Because scmutil.readrequires() was performing I/O and validation, we inlined the validation into localrepository.__init__ and removed scmutil.readrequires(). I plan to remove scmutil.readrequires() in a future commit (we can't do it now because statichttprepo uses it). Differential Revision: https://phab.mercurial-scm.org/D4568

localrepo: check for .hg/ directory in makelocalrepository() As part of this, we move the check to before .hg/hgrc is loaded, as it makes sense to check for the directory before attempting to open a file in it. Differential Revision: https://phab.mercurial-scm.org/D4567

localrepo: load extensions in makelocalrepository() Behavior does change subtly. First, we now load the hgrc before optionally setting up the vfs ward. That's fine: the vfs ward is for debugging and we know we won't hit it when reading .hg/hgrc. If the loaded extension were performing repo/vfs I/O, then we'd be worried. But extensions don't have access to the repo object that loaded them when they are loaded. Unless they are doing stack walking as part of module loading (which would be crazy), they shouldn't have access to the repo that incurred their load. Second, we now load extensions outside of the try..except IOError block. Previously, if loading an extension raised IOError, it would be silently ignored. I'm pretty sure the IOError is there for missing .hgrc files and should never have been ignored for issues loading extensions. I don't think this matters in reality because extension loading traps I/O errors. Differential Revision: https://phab.mercurial-scm.org/D4566

localrepo: copy ui in makelocalrepository() We will want to load the .hg/hgrc file from makelocalrepository() so we can consult its options as part of deriving the repository type. This means we need to create our ui instance copy in that function. Differential Revision: https://phab.mercurial-scm.org/D4565

localrepo: move some vfs initialization out of __init__ In order to make repository types more dynamic, we'll need to move the logic for determining repository behavior out of localrepository.__init__ so we can influence behavior before the type is instantiated. This commit starts that process by moving working directory and .hg/ vfs initialization to our new standalone function for instantiating local repositories. Aside from API changes, behavior should be fully backwards compatible. .. api:: localrepository.__init__ now does less work and accepts new args Use ``hg.repository()``, ``localrepo.instance()``, or ``localrepo.makelocalrepository()`` to obtain a new local repository instance instead of calling the ``localrepository`` constructor directly. Differential Revision: https://phab.mercurial-scm.org/D4564

localrepo: create new function for instantiating a local repo object Today, there is a single local repository class - localrepository. Its __init__ is responsible for loading the .hg/requires file and taking different actions depending on what is present. In addition, extensions may define a "reposetup" function that monkeypatches constructed repository instances, often by implementing a derived type and changing the __class__ of the repo instance. Work around alternate storage backends and partial clone has made it clear to me that shoehorning all this logic into __init__ and operating on an existing instance is too convoluted. For example, localrepository assumes revlog storage and swapping in non-revlog storage requires overriding e.g. file() to return something that isn't a revlog. I've authored various patches that either: a) teach various methods (like file()) about different states and taking the appropriate code path at run-time b) create methods/attributes/callables used for instantiating things and populating these in __init__ "a" incurs run-time performance penalties and makes code more complicated since various functions have a bunch of "if storage is X" branches. "b" makes localrepository quickly explode in complexity. My plan for tackling this problem is to make the local repository type more dynamic. Instead of a static localrepository class/type that supports all of the local repository configurations (revlogs vs other, revlogs with ellipsis, revlog v1 versus revlog v2, etc), we'll dynamically construct a type providing the implementations that are needed for the repository on disk, derived from the .hg/requires file and configuration options. The constructed repository type will be specialized and methods won't need to be taught about different implementations nor overloaded. We may also leverage this functionality for building types that don't implement all attributes. For example, the "intents" feature allows commands to declare that they are read only. By dynamically constructing a repository type, we could return a repository instance with no attributes related to mutating the repository. This could include things like a "changelog" property implementation that doesn't check whether it needs to invalidate the hidden revisions set on every access. This commit establishes a function for building a local repository instance. Future commits will start moving functionality from localrepository.__init__ to this function. Then we'll start dynamically changing the returned type depending on options that are present. This change may seem radical. But it should be fully compatible with the reposetup() model - at least for now. Differential Revision: https://phab.mercurial-scm.org/D4563

transaction: make entries a private attribute (API) This attribute is tracking changes to append-only files. It is an implementation detail and should not be exposed as part of the public interface. But code in repair was accessing it, so it seemingly does belong as part of the public API. But that code in repair is making assumptions about how storage works and is grossly wrong when alternate storage backends are in play. We'll need some kind of "strip" API at the storage layer that knows how to handle things in a storage-agnostic manner. I don't think accessing a private attribute on the transaction is any worse than what this code is already doing. So I'm fine with violating the abstraction for transactions. And with this change, all per-instance attributes on transaction have been made private except for "changes" and "hookargs." Both are used by multiple consumers and look like they need to be part of the public interface. .. api:: Various attributes of ``transaction.transaction`` are now ``_`` prefixed to indicate they shouldn't be used by external consumers. Differential Revision: https://phab.mercurial-scm.org/D4634

transaction: make names a private attribute This is used to report the transaction name in __repr__. It is very obviously an implementation detail and doesn't need to be exposed as part of the public interface. So mark it as private. Differential Revision: https://phab.mercurial-scm.org/D4633

transaction: make map a private attribute This is used to track which files are modified. It is an implementation detail of current transactions and doesn't need to be exposed to the public interface. So mark it as private. Differential Revision: https://phab.mercurial-scm.org/D4632

transaction: make report a private attribute This is a callable used for logging. It isn't used outside the transaction code. It doesn't need to be part of the public interface. Let's mark it as private. Differential Revision: https://phab.mercurial-scm.org/D4631

transaction: make opener a private attribute The VFS instance is an implementation detail of the transaction and doesn't belong as part of the public interface. So mark it as private. Differential Revision: https://phab.mercurial-scm.org/D4630

transaction: make after a private attribute This is another callable that is passed in at __init__ time. It doesn't need to be part of the public interface. Differential Revision: https://phab.mercurial-scm.org/D4629

transaction: make checkambigfiles a private attribute This holds instance state that is passed in at __init__ time. It doesn't need to be exposed as part of the public interface. Differential Revision: https://phab.mercurial-scm.org/D4628

transaction: make validator a private attribute This is similar to releasefn. It holds state that doesn't need to be exposed as part of the public interface. Differential Revision: https://phab.mercurial-scm.org/D4627

transaction: make releasefn a private attribute This is a handle on a callable that is called when the journal is closed. The value is specified at __init__ time. It doesn't need to be exposed on the public interface. So mark it as private. Differential Revision: https://phab.mercurial-scm.org/D4626

transaction: make file a private attribute This holds a file handle for the journal file. This file handle should not be touched outside the journal class and doesn't belong on the public interface. Differential Revision: https://phab.mercurial-scm.org/D4625

transaction: make journal a private attribute This attribute tracks the name of the journal file. It is an implementation detail of the current transaction and therefore shouldn't be exposed as part of the interface. Let's mark it as private. Differential Revision: https://phab.mercurial-scm.org/D4624

transaction: make undoname a private attribute This attribute tracks the file pattern to use for undo files. It is an implementation detail of the current transaction semantics and doesn't need to be part of the future transaction interface. So mark it as private. Differential Revision: https://phab.mercurial-scm.org/D4623

transaction: make count and usages private attributes I want to formalize the interface for transactions. As part of doing that, let's take the opportunity to make some attributes non-public. "count" and "usages" track how many times the transaction has been opened/nested/closed/released. This is internal state and doesn't need to be part of the public API. Differential Revision: https://phab.mercurial-scm.org/D4622

narrow: don't send the changelog information when widening without ellipses When we widen anon-ellipses narrow copy, the server sends the changelog information of all the changesets. The code was copied from ellipses case and in ellipses cases, it's required to send the new changelog data. But in non-ellipses cases, we don't need to send the changelog data as we will have all the changesets locally. Before this patch, there was a overhead of ~8-10 mins on each widening call because of all the changelog information being pulled and being applied. After this patch, we no more pull the changelog information. So this patch can save ~5 mins on Mozilla repo on each widening and more on repos which have more changesets. When we apply an empty changelog from changegroup, there is a devel-warn. This patch kind of hacks to silence that devel-warn. Differential Revision: https://phab.mercurial-scm.org/D4639

changegroup: add functionality to skip adding changelog data to changegroup In narrow extension, when we have a non-ellipses narrow working copy and we extend it, we pull all the changelog data again and the client tries to reapply all that changelog data. While downloading millions of changeset data is still not very expensive but applying them on the client side is very expensive and takes ~10 minutes. These 10 minutes are added to every `hg tracked --addinclude <>` call and extending a narrow copy becomes very slow. This patch adds a new changelog argument to cgpacker.generate() fn. If the changelog argument is set to False, we won't yield the changelog data. We still have to iterate over the deltas returned by _generatechangelog() because that's a generator and builds the data for clstate variable which is required for calculating manifests and filelogs. Differential Revision: https://phab.mercurial-scm.org/D4638

tests: add debug output in test-narrow-widen-no-ellipsis.t This will help us in understanding the upcoming patches better. Differential Revision: https://phab.mercurial-scm.org/D4637

changegroup: improve the devel-warn to specify changelog was empty Right now, the develwarn says "applied empty changegroup" which is not correct because we can send a changegroup without changelog with just manifest and filelogs and it will still say the same. Let's fix this to say that we are applying empty changelog from changegroup. In future patches I am will be adding functionality to send a changegroup from the server without an empty changelog. Differential Revision: https://phab.mercurial-scm.org/D4636

zsh_completion: add -b/--branch and -B/--bookmark(s) flags properly _hg_branch_bmark_opts used to add these two flags, but had the same descriptions for the flags regardless of what command took them and didn't allow specifying flags more than once (no '*' at the start). Even more importantly, it assumed that -B was always expecting an argument (i.e. --bookmark=foo), but in case of incoming and outgoing it's not so (--bookmarks is self-sufficient). Differential Revision: https://phab.mercurial-scm.org/D4612

narrow: when writing treemanifests, skip inspecting directories outside narrow This provides significant speed benefits when narrow and treemanifests are in use, see the timing numbers below. Note that like previously, differences of <5% are considered noise. The below timing numbers are in the same style as previously (example: ee7ee0c516ca). 'before' is 9db85644, and does not include that example commit's improvements. diff --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.327 s +- 0.051 s | 1.296 s +- 0.009 s | 97.7% m-u | | x | 1.310 s +- 0.020 s | 1.295 s +- 0.015 s | 98.9% m-u | x | | 1.295 s +- 0.018 s | 1.296 s +- 0.007 s | 100.1% m-u | x | x | 83.5 ms +- 0.8 ms | 84.1 ms +- 0.8 ms | 100.7% l-d-r | | | 205.1 ms +- 3.5 ms | 205.0 ms +- 3.8 ms | 100.0% l-d-r | | x | 194.2 ms +- 5.6 ms | 192.3 ms +- 4.3 ms | 99.0% l-d-r | x | | 99.1 ms +- 2.2 ms | 97.8 ms +- 0.9 ms | 98.7% l-d-r | x | x | 66.2 ms +- 1.0 ms | 67.2 ms +- 2.7 ms | 101.5% diff -c . --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 233.9 ms +- 1.9 ms | 235.6 ms +- 5.1 ms | 100.7% m-u | | x | 151.4 ms +- 1.2 ms | 152.2 ms +- 2.0 ms | 100.5% m-u | x | | 234.8 ms +- 2.7 ms | 235.0 ms +- 2.7 ms | 100.1% m-u | x | x | 127.8 ms +- 2.1 ms | 126.0 ms +- 1.1 ms | 98.6% l-d-r | | | 82.5 ms +- 1.6 ms | 82.3 ms +- 2.0 ms | 99.8% l-d-r | | x | 3.742 s +- 0.017 s | 3.819 s +- 0.208 s | 102.1% l-d-r | x | | 84.4 ms +- 1.5 ms | 83.2 ms +- 1.0 ms | 98.6% l-d-r | x | x | 751.2 ms +- 5.0 ms | 755.8 ms +- 12.9 ms | 100.6% rebase -r . --keep -d .^^: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 5.519 s +- 0.038 s | 5.526 s +- 0.057 s | 100.1% m-u | | x | 5.588 s +- 0.048 s | 5.607 s +- 0.061 s | 100.3% m-u | x | | 5.520 s +- 0.044 s | 5.546 s +- 0.059 s | 100.5% m-u | x | x | 586.6 ms +- 12.8 ms | 554.9 ms +- 21.2 ms | 94.6% <-- l-d-r | | | 629.8 ms +- 5.5 ms | 627.4 ms +- 6.6 ms | 99.6% l-d-r | | x | 6.165 s +- 0.058 s | 6.255 s +- 0.303 s | 101.5% l-d-r | x | | 270.2 ms +- 2.3 ms | 271.4 ms +- 2.7 ms | 100.4% l-d-r | x | x | 4.700 s +- 0.025 s | 1.651 s +- 0.016 s | 35.1% <-- status --change . --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 215.4 ms +- 2.3 ms | 216.5 ms +- 4.2 ms | 100.5% m-u | | x | 132.9 ms +- 1.2 ms | 132.0 ms +- 1.4 ms | 99.3% m-u | x | | 217.0 ms +- 1.9 ms | 215.4 ms +- 1.9 ms | 99.3% m-u | x | x | 108.6 ms +- 1.0 ms | 108.2 ms +- 1.5 ms | 99.6% l-d-r | | | 80.0 ms +- 1.3 ms | 80.5 ms +- 1.1 ms | 100.6% l-d-r | | x | 3.916 s +- 0.187 s | 3.966 s +- 0.236 s | 101.3% l-d-r | x | | 84.4 ms +- 3.1 ms | 83.9 ms +- 1.1 ms | 99.4% l-d-r | x | x | 758.0 ms +- 8.2 ms | 753.5 ms +- 5.0 ms | 99.4% status --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.905 s +- 0.025 s | 1.910 s +- 0.044 s | 100.3% m-u | | x | 1.892 s +- 0.009 s | 1.895 s +- 0.012 s | 100.2% m-u | x | | 1.891 s +- 0.012 s | 1.902 s +- 0.018 s | 100.6% m-u | x | x | 93.3 ms +- 0.9 ms | 93.4 ms +- 0.8 ms | 100.1% l-d-r | | | 570.7 ms +- 7.8 ms | 571.9 ms +- 18.5 ms | 100.2% l-d-r | | x | 561.5 ms +- 5.2 ms | 562.9 ms +- 6.1 ms | 100.2% l-d-r | x | | 171.7 ms +- 2.6 ms | 171.9 ms +- 1.2 ms | 100.1% l-d-r | x | x | 142.7 ms +- 2.0 ms | 140.3 ms +- 1.0 ms | 98.3% update $rev^; ~/src/hg/hg{hg}/hg update $rev: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 3.126 s +- 0.016 s | 3.128 s +- 0.015 s | 100.1% m-u | | x | 3.014 s +- 0.068 s | 3.008 s +- 0.031 s | 99.8% m-u | x | | 3.143 s +- 0.037 s | 3.184 s +- 0.086 s | 101.3% m-u | x | x | 308.0 ms +- 1.8 ms | 308.1 ms +- 5.7 ms | 100.0% l-d-r | | | 430.8 ms +- 4.5 ms | 436.4 ms +- 8.7 ms | 101.3% l-d-r | | x | 9.676 s +- 0.127 s | 9.945 s +- 0.272 s | 102.8% l-d-r | x | | 254.2 ms +- 3.3 ms | 255.7 ms +- 3.1 ms | 100.6% l-d-r | x | x | 1.571 s +- 0.030 s | 1.555 s +- 0.014 s | 99.0% Differential Revision: https://phab.mercurial-scm.org/D4606

tests: fix a couple of drawdag.py references Differential Revision: https://phab.mercurial-scm.org/D4635

py3: fix kwargs handling in hgext/fastannotate.py Differential Revision: https://phab.mercurial-scm.org/D4588

narrow: use diffmatcher to send only new filelogs in non-ellipses widening Before this patch, when we widen a non-ellipses narrow clone, we downloads all the filelogs matching the resulting new matcher. This is same as the ellipses case but can be improved because, we don't pull new csets in non-ellipses cases, we can only download the new added files instead of downloading all the files which matches the new matcher. So, we only download files which matches the new matcher but does not matches the old matcher. There exists a match.differencematcher() which is used here. This will lead to significant amount of speedup in extending a non-ellipses narrow copy on large repos because we will download and process only the new required filelogs. The tests changes demonstrate that we are downloading now less files. Thanks to Augie for pointing that functionality of differencematcher exists in core. Differential Revision: https://phab.mercurial-scm.org/D4614

py3: add missing b'' prefixes in couple of test files These were missed in the earlier patch and caught by Yuya. # skip-blame because just b'' prefix Differential Revision: https://phab.mercurial-scm.org/D4613

run-tests: convert the remaining os.system() call to Unicode I wasn't able to hit this path in 543a788eea2d, but I have now when I accidentally left off `--local`.

py3: partially fix pager spawning on Windows Previously, spinning up the pager crashed because the command and environment was in bytes. (See also 543a788eea2d.) Now it aborts with an invalid handle: $ HGMODULEPOLICY=py py -3 ../hg --traceback --config extensions.evolve=! Traceback (most recent call last): File "c:\Users\Matt\projects\hg\mercurial\ui.py", line 967, in _write self.fout.write(''.join(msgs)) File "c:\Users\Matt\projects\hg\mercurial\windows.py", line 173, in write self.fp.write(s[start:end]) OSError: [WinError 6] The handle is invalid During handling of the above exception, another exception occurred: Traceback (most recent call last): File "c:\Users\Matt\projects\hg\mercurial\scmutil.py", line 164, in callcatch return func() File "c:\Users\Matt\projects\hg\mercurial\dispatch.py", line 350, in _runcatchfunc return _dispatch(req) File "c:\Users\Matt\projects\hg\mercurial\dispatch.py", line 930, in _dispatch return commands.help_(ui, 'shortlist') File "c:\Users\Matt\projects\hg\mercurial\commands.py", line 2930, in help_ ui.write(formatted) File "c:\Users\Matt\projects\hg\mercurial\ui.py", line 948, in write self._writenobuf(*args, **opts) File "c:\Users\Matt\projects\hg\mercurial\ui.py", line 960, in _writenobuf self._write(*msgs, **opts) File "c:\Users\Matt\projects\hg\mercurial\ui.py", line 969, in _write raise error.StdioError(err) mercurial.error.StdioError: [Errno 9] The handle is invalid abort: The handle is invalid The interesting bit here is that the abort message is marked with ANSI color, but the OSError is not.

censor: rename loop variable to silence pyflakes warning hgext/censor.py:92: list comprehension redefines 'c' from line 88

py3: add b'' prefixes in tests/test-hgweb-no-request-uri.t # skip-blame because just b'' prefixes. Differential Revision: https://phab.mercurial-scm.org/D4611

py3: add b'' prefixes in tests/test-hgweb-no-path-info.t # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D4610

py3: add b'' prefixes in tests/test-hgweb-non-interactive.t # skip-blame because just b'' prefix Differential Revision: https://phab.mercurial-scm.org/D4609

py3: use codecs.encode() to encode in rot-13 encoding The other occurence will need some more love as description is bytes by default and we need to decode it and then encode it. Differential Revision: https://phab.mercurial-scm.org/D4608

py3: add two passing tests to whitelist found by buildbot The buildbot found these two new passing tests on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4607

phabricator: mark extension as experimental for now I don't want us to commit to this having a stable interface just yet. Differential Revision: https://phab.mercurial-scm.org/D4605

phabricator: fix templating bug by using hybriddict Differential Revision: https://phab.mercurial-scm.org/D4604

phabricator: add tests of templatekeyword Having tests is paying off: I found a bug and now it'll be easy to fix! Differential Revision: https://phab.mercurial-scm.org/D4603

phabricator: move extension from contrib to hgext It's well-enough tested now and widely enough used I think we should ship it. Differential Revision: https://phab.mercurial-scm.org/D4602

tests: add some basic tests of phabricator interactions This uses the vcr library to avoid hitting phabricator on every test execution. In order to generate new recordings (vcr calls them cassettes) just remove the appropriate json file, and the test will regenerate it. It's not my favorite way to test things, but it'll let us have test coverage on the phabricator extension that'll make it resilient to refactors in core and let us move it to hgext. In the future, it'd probably be better to have a docker container we can spin up for creating the vcr recordings, but for now this is enough better than nothing I'm going to declare victory. Coverage reports about 73% of the extension is now covered. Differential Revision: https://phab.mercurial-scm.org/D4601

phabricator: add support for using the vcr library to mock interactions I'll use this in an upcoming test. The decorator dancing in this is more complicated than I'd like, but it beats repeating all this code everywhere. Differential Revision: https://phab.mercurial-scm.org/D4600

keepalive: work around slight deficiency in vcr VCR's response type doesn't define the will_close attribute. Let's just have keepalive default to closing the socket if the will_close attribute is missing. Differential Revision: https://phab.mercurial-scm.org/D4599

hghave: add a checker for the vcr HTTP record/replay library I'm going to use this to write some tests of the phabricator extension. Differential Revision: https://phab.mercurial-scm.org/D4598

py3: allow run-tests.py to run on Windows This is now functional: HGMODULEPOLICY=py py -3 run-tests.py --local test-help.t --pure --view bcompare However, on this machine without a C compiler, it tries to load cext anyway, and blows up. I haven't looked into why, other than to see that it does set the environment variable. When the test exits though, I see it can't find killdaemons.py, get-with-headers.py, etc. I have no idea why these changes are needed, given that it runs on Linux. But os.system() is insisting that it take a str, and subprocess.Popen() blows up without str: Errored test-help.t: Traceback (most recent call last): File "run-tests.py", line 810, in run self.runTest() File "run-tests.py", line 858, in runTest ret, out = self._run(env) File "run-tests.py", line 1268, in _run exitcode, output = self._runcommand(cmd, env) File "run-tests.py", line 1141, in _runcommand env=env) File "C:\Program Files\Python37\lib\subprocess.py", line 756, in __init__ restore_signals, start_new_session) File "C:\Program Files\Python37\lib\subprocess.py", line 1100, in _execute_child args = list2cmdline(args) File "C:\Program Files\Python37\lib\subprocess.py", line 511, in list2cmdline needquote = (" " in arg) or ("\t" in arg) or not arg TypeError: argument of type 'int' is not iterable This is exactly how it crashes when trying to spin up a pager too. I left one instance of os.system() unchanged in _installhg(), because it doesn't get there.

py3: ensure run-tests environment is uniformly str subprocess.popen() was crashing, and when I printed out `env`, all of the keys and most of the values were str. Except these.

py3: ensure run-tests.osenvironb is actually bytes Windows doesn't have os.environb, so it was falling back to the Unicode form, and all of the accesses are trying to use bytes.

py3: fix str vs bytes in enough places to run `hg version` on Windows I don't have Visual Studio 2015 at home, but this now works with a handful of extensions (blackbox, extdiff, patchbomb, phabricator and rebase, but not evolve): $ HGMODULEPOLICY=py py -3 ../hg version Enabling the evolve extension causes the usual "failed to import ..." line, but then print this before the usual version output: ('commit', '[b'debugancestor', b'debugapplystreamclonebundle', ..., b'verify', b'version']') ... where the elided part seems to be every command and alias known.

windows: open registry keys using unicode names Python3 complained it must be str. While here, use a context manager to close the key- it wouldn't wrap at 80 characters the old way, and would have had to move anyway.

py3: byteify strings in pycompat These surfaced when disabling the source transformer to debug the problems in win32.py. ./contrib/byteify-strings.py found a couple false positives, so I marked them with r'' explicitly (in case I'm wrong). # skip-blame since this is just b'' and r'' prefixing

wireprotov2: let clients drive delta behavior Previously, the "manifestdata" and "filedata" commands assumed the receiver had all parent revisions for requested nodes. Unless the revision had no parents, they emitted a delta instead of a fulltext. This strategy isn't appropriate for shallow clones and for clients that only want to access fulltext revision data for a single node without fetching their parent revisions. This commit adds an "haveparents" argument to the "manifestdata" and "filedata" commands that controls delta generation behavior. Unless "haveparents" is set, the server assumes that the client doesn't have parent revisions unless they were previously sent as part of the current group of revisions. This change allows the fulltext revision data of any individual revision to be obtained. This will facilitate shallow clones and other data retrieval strategies that don't require all previous revisions of an entity to be fetched. Differential Revision: https://phab.mercurial-scm.org/D4492

exchangev2: fetch file revisions Now that the server has an API for fetching file data, we can call into it to fetch file revisions. The implementation is relatively straightforward: we examine the manifests that we fetched and find all new file revisions referenced by them. We build up a mapping from file path to file nodes to manifest node. (The mapping to first manifest node allows us to map back to first changelog node/revision, which is used for the linkrev.) Once that map is built up, we iterate over it in a deterministic manner and fetch and store file data. The code is very similar to manifest fetching. So similar that we could probably extract the common bits into a generic function. With file data retrieval implemented, `hg clone` and `hg pull` are effectively feature complete, at least as far as the completeness of data transfer for essential repository data (changesets, manifests, files, phases, and bookmarks). We're still missing support for obsolescence markers, the hgtags fnodes cache, and the branchmap cache. But these are non-essential for the moment (and will be implemented later). This is a good point to assess the state of exchangev2 in terms of performance. I ran a local `hg clone` for the mozilla-unified repository using both version 1 and version 2 of the wire protocols and exchange methods. This is effectively comparing the performance of the wire protocol overhead and "getbundle" versus domain-specific commands. Wire protocol version 2 doesn't have compression implemented yet. So I tested version 1 with `server.compressionengines=none` to remove compression overhead from the equation. server before: user 220.420+0.000 sys 14.420+0.000 after: user 321.980+0.000 sys 18.990+0.000 client before: real 561.650 secs (user 497.670+0.000 sys 28.160+0.000) after: real 1226.260 secs (user 944.240+0.000 sys 354.150+0.000) We have substantial regressions on both client and server. This is obviously not desirable. I'm aware of some reasons: * Lack of hgtagsfnodes transfer (contributes significant CPU to client). * Lack of branch cache transfer (contributes significant CPU to client). * Little to no profiling / optimization performed on wire protocol version 2 code. * There appears to be a memory leak on the client and that is likely causing swapping on my machine. * Using multiple threads on the client may be counter-productive because Python. * We're not compressing on the server. * We're tracking file nodes on the client via manifest diffing rather than using linkrev shortcuts on the server. I'm pretty confident that most of these issues are addressable. But even if we can't get wire protocol version 2 on performance parity with "getbundle," I still think it is important to have the set of low level data-specific retrieval commands that we have implemented so far. This is because the existence of such commands allows flexibility in how clients access server data. Differential Revision: https://phab.mercurial-scm.org/D4491

wireprotov2: define and implement "filedata" command Continuing our trend of implementing *data commands for retrieving information about specific repository data primitives, this commit implements a command for retrieving data about an individual tracked file. The command is very similar to "manifestdata." The only significant difference is that we have a standalone function for obtaining storage for a tracked file. This is to provide a monkeypatch point for extensions to implement path-based access control. With this API available, wire protocol version 2 now exposes all data primitives necessary to implement a full clone. Of course, since "filedata" can only resolve data for a single path at a time, clients would need to issue N commands to perform a full clone. On the Firefox repository, this would be ~461k commands. We'll likely need to implement a file data retrieval command that supports multiple paths. But that can be implemented later. Differential Revision: https://phab.mercurial-scm.org/D4490

exchangev2: fetch manifest revisions Now that the server has support for retrieving manifest data, we can implement the client bits to call it. We teach the changeset fetching code to capture the manifest revisions that are encountered on incoming changesets. We then feed this into a new function which filters out known manifests and then batches up manifest data requests to the server. This is different from the previous wire protocol in a few notable ways. First, the client fetches manifest data separately and explicitly. Before, we'd ask the server for data pertaining to some changesets (via a "getbundle" command) and manifests (and files) would be sent automatically. Providing an API for looking up just manifest data separately gives clients much more flexibility for manifest management. For example, a client may choose to only fetch manifest data on demand instead of prefetching it (i.e. partial clone). Second, we send N commands to the server for manifest retrieval instead of 1. This property has a few nice side-effects. One is that the deterministic nature of the requests lends itself to server-side caching. For example, say the remote has 50,000 manifests. If the server is configured to cache responses, each time a new commit arrives, you will have a cache miss and need to regenerate all outgoing data. But if you makes N requests requesting 10,000 manifests each, a new commit will still yield cache hits on the initial, unchanged manifest batches/requests. A derived benefit from these properties is that resumable clone is conceptually simpler to implement. When making a monolithic request for all of the repository data, recovering from an interrupted clone is hard because the server was in the driver's seat and was maintaining state about all the data that needed transferred. With the client driving fetching, the client can persist the set of unfetched entities and retry/resume a fetch if something goes wrong. Or we can fetch all data N changesets at a time and slowly build up a repository. This approach is drastically easier to implement when we have server APIs exposing low-level repository primitives (such as manifests and files). We don't yet support tree manifests. But it should be possible to implement that with the existing wire protocol command. Differential Revision: https://phab.mercurial-scm.org/D4489

wireprotov2: define and implement "manifestdata" command The added command can be used for obtaining manifest data. Given a manifest path and set of manifest nodes, data about manifests can be retrieved. Unlike changeset data, we wish to emit deltas to describe manifest revisions. So the command uses the relatively new API for building delta requests and emitting them. The code calls into deltaparent(), which I'm not very keen of. There's still work to be done in delta generation land so implementation details of storage (e.g. exactly one delta is stored/available) don't creep into higher levels. But we can worry about this later (there is already a TODO on imanifestorage tracking this). On the subject of parent deltas, the server assumes parent revisions exist on the receiving end. This is obviously wrong for shallow clone. I've added TODOs to add a mechanism to the command to allow clients to specify desired behavior. This shouldn't be too difficult to implement. Another big change is that the client must explicitly request manifest nodes to retrieve. This is a major departure from "getbundle," where the server derives relevant manifests as it iterates changesets and sends them automatically. As implemented, the client must transmit each requested node to the server. At 20 bytes per node, we're looking at 2 MB per 100,000 nodes. Plus wire encoding overhead. This isn't ideal for clients with limited upload bandwidth. I plan to address this in the future by allowing alternate mechanisms for defining the revisions to retrieve. One idea is to define a range of changeset revisions whose manifest revisions to retrieve (similar to how "changesetdata" works). We almost certainly want an API to look up an individual manifest by node. And that's where I've chosen to start with the implementation. Again, a theme of this early exchangev2 work is I want to start by building primitives for accessing raw repository data first and see how far we can get with those before we need more complexity. Differential Revision: https://phab.mercurial-scm.org/D4488

wireprotov2: add TODOs around extending changesetdata fields Extensions will inevitably want to extend the set of changeset data/fields that can be requested. We'll need to implement support for extending this in the future. Add some TODOs to track that. Differential Revision: https://phab.mercurial-scm.org/D4487

exchangev2: fetch and apply bookmarks This is pretty similar to phases data. We collect bookmarks data as we process records. Then at the end we make a call to the bookmarks subsystem to reflect the remote's bookmarks. Like phases, the code for handling bookmarks is vastly simpler than the previous wire protocol code because the server always transfers the full set of bookmarks when bookmarks are requested. We don't have to keep track of whether we requested bookmarks or not. Differential Revision: https://phab.mercurial-scm.org/D4486

wireprotov2: add bookmarks to "changesetdata" command Like we did for phases, we want to emit bookmarks data attached to each changeset. The approach here is very similar to phases: we emit bookmarks data inline with requested revision data. But we emit records for nodes that weren't requested as well so consumers have access to the full set of defined bookmarks. Differential Revision: https://phab.mercurial-scm.org/D4485

exchangev2: fetch and apply phases data Now that the server supports emitting phases data, we can request it and apply it on the client. Because we may receive phases-only updates from the server, we no longer conditionally perform the "changesetdata" command depending on whether there are revisions to fetch. In the previous wire protocol, this case would result in us falling back to performing "listkeys" commands to look up phases, bookmarks, etc data. But since "changesetdata" is smart enough to handle metadata only fetches, we can keep things consistent. It's worth noting that because of the unified approach to changeset data retrieval, phase handling code in wire proto v2 exchange is drastically simpler. Contrast with all the code in exchange.py dealing with all the variations for obtaining phases data. Differential Revision: https://phab.mercurial-scm.org/D4484

wireprotov2: add phases to "changesetdata" command This commit teaches the "changesetdata" wire protocol command to emit the phase state for each changeset. This is a different approach from existing phase transfer in a few ways. Previously, if there are no new revisions (or we're not using bundle2), we perform a "listkeys" request to retrieve phase heads. And when revision data is being transferred with bundle2, phases data is encoded in a standalone bundle2 part. In both cases, phases data is logically decoupled from the changeset data and is encountered/applied after changeset revision data is received. The new wire protocol purposefully tries to more tightly associate changeset metadata (phases, bookmarks, obsolescence markers, etc) with the changeset revision and index data itself, rather than have it live as a separate entity that must be fetched and processed separately. I reckon that one reason we didn't do this before was it was difficult to add new data types/fields without breaking existing consumers. By using CBOR maps to transfer changeset data and putting clients in control of what fields are requested / present in those maps, we can easily add additional changeset data while maintaining backwards compatibility. I believe this to be a superior approach to the problem. That being said, for performance reasons, we may need to resort to alternative mechanisms for transferring data like phases. But for now, I think giving the wire protocol the ability to transfer changeset metadata next to the changeset itself is a powerful feature because it is a raw, changeset-centric data API. And if you build simple APIs for accessing the fundamental units of repository data, you enable client-side experimentation (partial clone, etc). If it turns out that we need specialized APIs or mechanisms for transferring data like phases, we can build in those APIs later. For now, I'd like to see how far we can get on simple APIs. It's worth noting that when phase data is being requested, the server will also emit changeset records for nodes in the bases specified by the "noderange" argument. This is to ensure that phase-only updates for nodes the client has are available to the client, even if no new changesets will be transferred. Differential Revision: https://phab.mercurial-scm.org/D4483

exchangev2: fetch changeset revisions All Mercurial repository data is derived from changesets: you can't do anything unless you have changesets. Therefore, it makes sense for changesets to be the first piece of data that we transfer as part of pull. To do this, we call our new "changesetdata" command, requesting parents and revision data. This gives us all the data that a changegroup delta group would give us. We simply normalize this data into what addgroup() expects and call that API on the changelog to bulk insert revisions into the changelog. Code in this commit is heavily borrowed from changegroup.cg1unpacker.apply(). Differential Revision: https://phab.mercurial-scm.org/D4482

wireprotov2: define and implement "changesetdata" command This commit introduces the "changesetdata" wire protocol command. The role of the command is to expose data associated with changelog revisions, including the raw revision data itself. This command is the first piece of a new clone/pull strategy that is built on top of domain-specific commands for data retrieval. Instead of a monolithic "getbundle" command that transfers all of the things, we'll be introducing commands for fetching specific pieces of data. Since the changeset is the fundamental unit from which we derive pointers to other data (manifests, file nodes, etc), it makes sense to start reimplementing pull with this data. The command accepts as arguments a set of root and head revisions defining the changesets that should be fetched as well as an explicit list of nodes. By default, the command returns only the node values: the client must explicitly request additional fields be added to the response. Current supported fields are the list of parent nodes and the revision fulltext. My plan is to eventually add support for transferring other data associated with changesets, including phases, bookmarks, obsolescence markers, etc. Since the response format is CBOR, we'll be able to add this data into the response object relatively easily (it should be as simple as adding a key in a map). The documentation captures a number of TODO items. Some of these may require BC breaking changes. That's fine: wire protocol v2 is still highly experimental. Differential Revision: https://phab.mercurial-scm.org/D4481

exchangev2: start to implement pull with wire protocol v2 Wire protocol version 2 will take a substantially different approach to exchange than version 1 (at least as far as pulling is concerned). This commit establishes a new exchangev2 module for holding code related to exchange using wire protocol v2. I could have added things to the existing exchange module. But it is already quite big. And doing things inline isn't in question because the existing code is already littered with conditional code for various states of support for the existing wire protocol as it evolved over 10+ years. A new module gives us a chance to make a clean break. This approach does mean we'll end up writing some duplicate code. And there's a significant chance we'll miss functionality as code is ported. The plan is to eventually add #testcase's to existing tests so the new wire protocol is tested side-by-side with the existing one. This will hopefully tease out any features that weren't ported properly. But before we get there, we need to build up support for the new exchange methods. Our journey towards implementing a new exchange begins with pulling. And pulling begins with discovery. The discovery code added to exchangev2 is heavily drawn from the following functions: * exchange._pulldiscoverychangegroup * discovery.findcommonincoming For now, we build on top of existing discovery mechanisms. The new wire protocol should be capable of doing things more efficiently. But I'd rather defer on this problem. To foster the transition, we invent a fake capability on the HTTPv2 peer and have the main pull code in exchange.py call into exchangev2 when the new wire protocol is being used. Differential Revision: https://phab.mercurial-scm.org/D4480

httppeer: expose capabilities for each command This will help code using peers to sniff out exactly what servers support. Differential Revision: https://phab.mercurial-scm.org/D4436

narrow: intersect provided matcher with narrowmatcher in `hg diff` This provides significant speedups when running diff, and no change in behavior that I'm aware of (or that the tests found). I tested with a repo that I started using narrow in after it was created and attempted to run `hg diff -c .` and similar commands in it on a commit that had files not in the narrowspec. Timing numbers below, using a similar setup as my previous commits. before=9db85644, m-u is mozilla-unified at eb39298e432d (flatmanifest) and 0553b7f29eaf (treemanifest). l-d-r is a repo simulating a situation I've encountered where there's one directory with 30k+ subdirectories. N means narrow, T means treemanifest. The narrowspec is pretty small when in use, and importantly the narrowspec is applied *after* doing the initial checkout (without narrowing), so all of these files exist in the filesystem, which is not normally the case if someone has been using narrow for the entire life of the clone. Anything less than a 5% difference in performance is most likely noise. diff --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.292 s +- 0.009 s | 1.295 s +- 0.010 s | 100.2% m-u | | x | 1.296 s +- 0.042 s | 1.299 s +- 0.026 s | 100.2% m-u | x | | 1.292 s +- 0.010 s | 1.297 s +- 0.021 s | 100.4% m-u | x | x | 84.2 ms +- 1.2 ms | 83.6 ms +- 0.2 ms | 99.3% l-d-r | | | 188.7 ms +- 2.7 ms | 188.8 ms +- 2.0 ms | 100.1% l-d-r | | x | 189.9 ms +- 1.5 ms | 189.4 ms +- 1.2 ms | 99.7% l-d-r | x | | 97.1 ms +- 1.0 ms | 87.1 ms +- 1.0 ms | 89.7% <-- l-d-r | x | x | 96.9 ms +- 0.8 ms | 87.2 ms +- 0.7 ms | 90.0% <-- diff -c . --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 231.6 ms +- 3.1 ms | 228.9 ms +- 1.6 ms | 98.8% m-u | | x | 150.5 ms +- 1.7 ms | 150.7 ms +- 1.4 ms | 100.1% m-u | x | | 233.7 ms +- 2.4 ms | 232.2 ms +- 1.9 ms | 99.4% m-u | x | x | 126.1 ms +- 1.2 ms | 126.8 ms +- 1.2 ms | 100.6% l-d-r | | | 82.1 ms +- 2.0 ms | 81.8 ms +- 1.4 ms | 99.6% l-d-r | | x | 3.732 s +- 0.020 s | 3.746 s +- 0.027 s | 100.4% l-d-r | x | | 83.1 ms +- 0.8 ms | 107.6 ms +- 2.4 ms | 129.5% <-- l-d-r | x | x | 758.2 ms +- 38.8 ms | 188.5 ms +- 1.8 ms | 24.9% <-- rebase -r . --keep -d .^^: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 5.532 s +- 0.087 s | 5.496 s +- 0.016 s | 99.3% m-u | | x | 5.554 s +- 0.061 s | 5.532 s +- 0.013 s | 99.6% m-u | x | | 5.602 s +- 0.134 s | 5.508 s +- 0.035 s | 98.3% m-u | x | x | 582.2 ms +- 15.2 ms | 572.9 ms +- 12.0 ms | 98.4% l-d-r | | | 629.5 ms +- 12.3 ms | 622.5 ms +- 7.3 ms | 98.9% l-d-r | | x | 6.173 s +- 0.062 s | 6.185 s +- 0.076 s | 100.2% l-d-r | x | | 274.5 ms +- 10.0 ms | 272.1 ms +- 6.2 ms | 99.1% l-d-r | x | x | 4.835 s +- 0.056 s | 4.826 s +- 0.034 s | 99.8% status --change . --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 214.4 ms +- 1.4 ms | 212.2 ms +- 1.7 ms | 99.0% m-u | | x | 130.9 ms +- 1.2 ms | 131.7 ms +- 1.1 ms | 100.6% m-u | x | | 215.0 ms +- 2.1 ms | 214.9 ms +- 2.7 ms | 100.0% m-u | x | x | 109.5 ms +- 2.3 ms | 107.8 ms +- 0.9 ms | 98.4% l-d-r | | | 79.6 ms +- 0.9 ms | 79.8 ms +- 1.6 ms | 100.3% l-d-r | | x | 3.799 s +- 0.037 s | 3.928 s +- 0.021 s | 103.4% <--? l-d-r | x | | 82.7 ms +- 0.7 ms | 83.2 ms +- 1.0 ms | 100.6% l-d-r | x | x | 746.8 ms +- 6.1 ms | 739.0 ms +- 4.2 ms | 99.0% status --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.884 s +- 0.012 s | 1.885 s +- 0.013 s | 100.1% m-u | | x | 1.897 s +- 0.027 s | 1.909 s +- 0.077 s | 100.6% m-u | x | | 1.886 s +- 0.021 s | 1.891 s +- 0.030 s | 100.3% m-u | x | x | 92.0 ms +- 0.7 ms | 92.4 ms +- 0.4 ms | 100.4% l-d-r | | | 570.3 ms +- 18.7 ms | 552.2 ms +- 4.5 ms | 96.8% l-d-r | | x | 568.9 ms +- 16.1 ms | 567.2 ms +- 11.9 ms | 99.7% l-d-r | x | | 171.1 ms +- 2.5 ms | 170.4 ms +- 1.2 ms | 99.6% l-d-r | x | x | 171.6 ms +- 3.4 ms | 171.5 ms +- 1.7 ms | 99.9% update $rev^; ~/src/hg/hg{hg}/hg update $rev: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 3.107 s +- 0.017 s | 3.116 s +- 0.012 s | 100.3% m-u | | x | 2.943 s +- 0.010 s | 2.945 s +- 0.019 s | 100.1% m-u | x | | 3.116 s +- 0.033 s | 3.118 s +- 0.027 s | 100.1% m-u | x | x | 318.5 ms +- 2.7 ms | 320.8 ms +- 4.8 ms | 100.7% l-d-r | | | 428.9 ms +- 4.4 ms | 429.5 ms +- 4.0 ms | 100.1% l-d-r | | x | 9.593 s +- 0.081 s | 9.869 s +- 0.043 s | 102.9% l-d-r | x | | 253.2 ms +- 3.6 ms | 254.0 ms +- 2.8 ms | 100.3% l-d-r | x | x | 1.613 s +- 0.009 s | 1.630 s +- 0.017 s | 101.1% Differential Revision: https://phab.mercurial-scm.org/D4587

identify: change {parents} to a list of nodes (BC) This is a part of the name unification. {parents} is a list of nodes in "hg log -Tjson" output. Since {rev} can be computed from (repo, node) pair, we no longer need to put it to provide {rev} to user templates. https://www.mercurial-scm.org/wiki/GenericTemplatingPlan#Dictionary

identify: use fm.hexfunc thoroughly This fixes the length of {id} in JSON and template outputs.

formatter: replace contexthint() with demand loading of ctx object And pass in repo instead to resolve ctx from (repo, node) pair.

formatter: populate ctx from repo and node value This will basically replace the fm.contexthint() API. I originally thought this would be too complicated, and I wrote 8399438bc7ef "formatter: provide hint of context keys required by template" because of that. However, I had to add a similar mechanism for fctx templates, and the overall machinery became way simpler than my original patch. The test output slightly changed as {author} is no longer available in the {manifest} context, which isn't the point this test targeted on.

merge with stable

py3: call hgweb.hgweb() with bytes values # skip-blame because just b'' prefixes I believe this should fix some tests. Differential Revision: https://phab.mercurial-scm.org/D4594

py3: use '%d' for integers instead of '%s' Differential Revision: https://phab.mercurial-scm.org/D4593

py3: use "%f" for floats instead of "%s" Differential Revision: https://phab.mercurial-scm.org/D4592

py3: suppress the return value from .write() call Differential Revision: https://phab.mercurial-scm.org/D4591

py3: add b'' prefixes in tests/test-diff-color.t # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D4590

py3: slice through bytes to prevent getting ascii value I still don't know why python-dev thought it was a nice idea to do this. Differential Revision: https://phab.mercurial-scm.org/D4589

censor: use a reasonable amount of memory Before this change, trying to censor some random revision uses an ever increasing amount of memory (I stopped at 20GB, but it was by no means finished), presumably because these contexts have a lot of information that is kept alive. After this change, the memory usage plateaus quickly. Differential Revision: https://phab.mercurial-scm.org/D4582

help: add internals.wireprotocolrpc to the table

setup: exclude vendored futures package on Python 3 The vendored future can't live on Python 3.

py3: whitelist another passing test Differential Revision: https://phab.mercurial-scm.org/D4562

py3: prevent the win32 ctype _fields_ from being transformed to bytes Otherwise, any hg invocation dies with TypeError: '_fields_' must be a sequence of (name, C type) pairs # skip-blame just a r prefix

cext: fix warnings when building for py3 on Windows MSVC++ 14 now has standard int types that don't need to be redefined (I didn't go back to see when they came along since the build system wants either 2008 or 2015), but doesn't have ssize_t. The FILE pointer in posixfile is only used on python2.

cext: stop preprocessing a partial function call MSVC++ 14 yelled: mercurial/cext/revlog.c(1913): fatal error C1057: unexpected end of file in macro expansion At this point, the C extensions build (with warnings), and it dies in win32.py because the `_fields_` strings in the ctypes classes are being converted to bytes by the source translator.

py3: add b'' to some setup.py strings for Windows These were things found trying to do `make PYTHON="py -3" local`. The following is dumped out, before dying while compiling the C extensions: C:\Program Files\Python37\lib\site-packages\setuptools\dist.py:406: UserWarning: The version specified (b'4.7.1') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details. "details." % self.metadata.version running build_py byte-compiling .\mercurial\thirdparty\concurrent\futures\_base.py to _base.cpython-37.pyc File "mercurial\thirdparty\concurrent\futures\_base.py", line 416 raise exception_type, self._exception, self._traceback ^ SyntaxError: invalid syntax # skip-blame since these are just converting to bytes literals

dagop: fix typo spotted while doing unrelated investigation Differential Revision: https://phab.mercurial-scm.org/D4584

hg: don't reuse repo instance after unshare() Unsharing a repository is a pretty invasive procedure and fundamentally changes the behavior of the repository. Currently, hg.unshare() calls into localrepository.__init__ to re-initialize the repository instance. This is a bit hacky. And future commits that refactor how localrepository instances are constructed will make this difficult to support. This commit changes unshare() so it constructs a new repo instance once the unshare I/O has completed. It then poisons the old repo instance so any further use will result in error. Surprisingly, nothing in core appears to access a repo instance after it has been unshared! .. api:: ``hg.unshare()`` now poisons the repo instance so it can't be used. It also returns a new repo instance suitable for interacting with the unshared repository. Differential Revision: https://phab.mercurial-scm.org/D4557

unionrepo: dynamically create repository type from base repository This is basically the same thing we just did for bundlerepo except for union repositories. .. api:: ``unionrepo.unionrepository()`` is no longer usable on its own. To instantiate an instance, call ``unionrepo.instance()`` or ``unionrepo.makeunionrepository()``. Differential Revision: https://phab.mercurial-scm.org/D4556

bundlerepo: dynamically create repository type from base repository Previously, bundlerepository inherited from localrepo.localrepository. You simply instantiated a bundlerepository and its __init__ called localrepo.localrepository.__init__. Things were simple. Unfortunately, this strategy is limiting because it assumes that the base repository is a localrepository instance. And it assumes various properties of localrepository, such as the arguments its __init__ takes. And it prevents us from changing behavior of localrepository.__init__ without also having to change derived classes. Previous and ongoing work to abstract storage revealed these limitations. This commit changes the initialization strategy of bundle repositories to dynamically create a type to represent the repository. Instead of a static type, we instantiate a new local repo instance via localrepo.instance(). We then combine its __class__ with bundlerepository to produce a new type. This ensures that no matter how localrepo.instance() decides to create a repository object, we can derive a bundle repo object from it. i.e. localrepo.instance() could return a type that isn't a localrepository and it would "just work." Well, it would "just work" if bundlerepository's custom implementations only accessed attributes in the documented repository interface. I'm pretty sure it violates the interface contract in a handful of places. But we can worry about that another day. This change gets us closer to doing more clever things around instantiating repository instances without having to worry about teaching bundlerepository about them. .. api:: ``bundlerepo.bundlerepository`` is no longer usable on its own. The class is combined with the class of the base repository it is associated with at run-time. New bundlerepository instances can be obtained by calling ``bundlerepo.instance()`` or ``bundlerepo.makebundlerepository()``. Differential Revision: https://phab.mercurial-scm.org/D4555

bundlerepo: factor out code for instantiating a bundle repository This code will soon become a bit more complicated. So extract to its own function. And change both instantiators of bundlerepository to use it. Differential Revision: https://phab.mercurial-scm.org/D4554

bundlerepo: pass create=True I don't want to know how this came to be. Maybe a holdover from the days before Python had a bool type? Differential Revision: https://phab.mercurial-scm.org/D4553

shelve: use bundlerepo.instance() to construct a repo object The instance() functions are preferred over cls.__init__ for creating repo instances. It doesn't really matter now. But future commits will refactor the bundlerepository class in ways that will cause the old way to break. Differential Revision: https://phab.mercurial-scm.org/D4552

templatekw: add experimental {status} keyword This is another example of fctx-based keywords. I think this is somewhat useful in log templates.

templatekw: add option to include ignored/clean/unknown files in cache They will be necessary to provide {status} of files.

templatekw: keep status tuple in cache dict and rename cache key accordingly There's no point to drop tail elements, which are mostly empty lists.

templatekw: extract function that computes and caches file status

py3: use sysstr() to convert ProgrammingError bytes with no unicode error risk msg.decode('utf8') may fail if msg isn't an ASCII string, and that's possible as we sometimes embed a filename in the error message for example.

revlog: reuse cached delta for identical base revision (issue5975) Since 8f83a953dddf, we skip over empty deltas when choosing a delta base. Such delta happens when two distinct revisions have the same content. The remote might be sending a delta against such revision within the bundle. In that case, the delta base is no longer considered, but the cached one could still, be used with the equivalent revision. Not reusing the delta from the bundle can have a significant performance impact, so we now make sure with doing so when possible.

snapshot: fix line order when skipping over empty deltas The code movement in 37957e07138c introduced an error. Since 8f83a953dddf, we discarded some revisions because they are identical to their delta base (and use that delta base instead). That logic is good, however, in 37957e07138c we mixed up the order of two line, adding the "new" revision to the set of already tested one, instead of the discarded one. So in practice, we were never investigating any revisions in a chain starting with an empty delta. Creating significantly worst delta chain (eg: Mercurial's manifest move goes from about 60MB up to about 80MB).

tests: stabilize change for handling not quoting non-empty-directory The change originated in cb1329738d64. I suspect the problem is with the combination of (re) and the '\' to '/' retry on Windows. I've no idea if py3 on Windows needs the quoting, since it can't even run `hg` with no arguments. (It's dying somewhere on the ctype declarations when win32.py is imported.)

hg: wrap the highest layer in the `hg` script possible in trace event This should help us have a better idea of what "interpreter startup costs" look like. This does omit the HGUNICODEPEDANTRY block and the LIBDIR dancing to set up sys.path, but the former is usually off and the latter is unavoidable and should be very fast. If we get worried about those cases we can consider open-coding the tracing logic here. Differential Revision: https://phab.mercurial-scm.org/D4346

localrepo: use urllocalpath() for path to create repo too It looks like this was lost in 7ce9dea3a14a (localrepo: move repo creation logic out of localrepository.__init__ (API), 2018-09-11). I don't know when it makes a difference (maybe on Windows, since urllocalpath() mentions something about drive letters). Differential Revision: https://phab.mercurial-scm.org/D4550

localrepo: move check for existing repo into createrepository() For symmetry with the check for existence of a repo in localrepository.__init__, we should check for the non-existence in createrepository(). We could alternatively move both checks into instance(). Differential Revision: https://phab.mercurial-scm.org/D4549

py3: add b'' to some run-tests.py strings for Windows Things go seriously off the rails after this, so there may be more that are missing. # skip-blame since these are just converting to bytes literals

wireprotov1peer: forward __name__ of wrapped method in batchable decorator Not required, but clarifies debugging when the going gets really tough. Differential Revision: https://phab.mercurial-scm.org/D4551

templatekw: add {size} keyword as an example of fctx-based keyword I'll add {status}, and I think some lfs keywords can be migrated to this. I'm not certain how many fctx-based keywords will be introduced into the global space, but if there are a couple more, we'll probably need to sort them out to the "File Keywords" section in the templater help. Until then, fctx keywords are hidden as experimental.

formatter: populate fctx from ctx and path value Tests will be added by the next patch.