wireprotov2peer: wait for initial object before resolving future As part of rolling out wireprotov2 with redirect support, I encountered an edge case with regards to future resolution. Essentially, the initial response frame from the server did not fully decode the initial CBOR object. The frame wasn't marked as EOS. In the previous code, we resolved the future for the request to response.objects(), which mapped to the commandresponse instance which would eventually produce a redirect. Upon receiving subsequent data, the initial CBOR object containing the redirect would be decoded and we'd process the redirect. However, the future would already have been resolved with the initial commandresponse.objects() and the client iterating over the objects wouldn't receive any objects from the redirect because the redirect was populating a different commandresponse instance! This commit changes the logic so we don't resolve futures until the initial CBOR response object is fully decoded or until EOS occurs. In cases where there is an empty or partial frame associated with a redirect, the future will now resolve with the commandresponse containing the proper series of decoded objects.

wireprotov2peer: always return a bool from _processredirect() Without this, we may stop servicing the redirect response if the future has already been resolved. And the future will often be resolved very early, since many consumers iterate the decoded CBOR object stream and expect data to lazily arrive.

tests: disable remotefilelog on Windows I've spent a non trivial amount of time trying to eliminate the test errors, but it's looking like this is pretty dependent on Unix support. For example, there are attempts to delete open files, and uses of threads that report I/O attempts on closed files. (Maybe this is a race condition? Don't we usually use processes as workers on Windows?) In any event, I don't want real new errors elsewhere to be masked by these known problems. For some reason $CACHEDIR is reported as missing in test-remotefilelog-repack.t, but it actually exists in the hgcloneshallow call inside shallowutil.mkstickygroupdir(). By the time the process exits, it's gone. I don't see it being removed by code that calls 'rmdir' or 'remove' in the extension itself.

perf: run 'setup' function during stub run The benchmarked function might need the content of the setup to be run in order to function properly.

perf: fallback to subset if ondisk cache is missing in perfbranchmapload If there is no branchmap on disk for that filter, it means that the cache from some subset's filter is relevant for this one. We look for it instead of aborting. That way it is much simpler to run the command in an automated way. We can now add it to `test-contrib-perf.t`.

perf: prewarm the branchmap in perfbranchmapload It is not very interesting to have the command randomly failing because the branchmap for the tested filter happens to be cold. So we make sure to have a valid up to date branchmap before going further. The data might still be missing from disk if a subset was equivalent. See next changeset for details and fix.

localrepo: correct docstring of filectx() The same reason as b6c2543e1dd8. It can't be any changeset specifiers but revision number.

annotate: remove dead code to not convert path to relative path It's annotate. There should be at least one file path specified.

annotate: start pager soon after command options are validated It helps extracting helper class.

annotate: check and update command options first It helps extracting helper class.

annotate: format local tables in less-dense form I'll move these to a class, where one-extra indent will be needed.

perf: rename version flag of perfchangegroupchangelog to cgversion The version flag conflicts with the global version flag and does not work. Differential Revision: https://phab.mercurial-scm.org/D5297 Differential Revision: https://phab.mercurial-scm.org/D5298

procutil: don't allow the main 'hg' script to be treated as the Windows exe Previously, there were a handful of errors like this: $ hg prefetch --repack (running background incremental repack) + abort: %1 is not a valid Win32 application + [255] CreateProcess() doesn't append .exe when `lpApplicationName` contains a path, and a python script isn't directly executable.

remotefilelog: drop some compatibility cruft for finding the hg exeutable

branchmap: build the revbranchcache._namesreverse() only when required On big repositories with a lot of named branches and that also increasing over time, building of this dict can be expensive and shows up in profile. For our internal repository, this saves ~0.05 seconds. Thanks to Yuya for suggesting using util.propertycache() and util.clearcachedproperty(). Differential Revision: https://phab.mercurial-scm.org/D5291

sparse-revlog: put the native implementation of slicechunktodensity to use When possible, the C implementation of delta chain slicing will be used. providing a large boost in performance for this operation. To take a practical example of restoring manifest revision '59547c40bc4c' for a reference NetBeans repository (using sparse-revlog). The media time of the step `slice-sparse-chain` of `perfrevlogrevision` improve from 0.660 ms to 0.098 ms; The full series move delta chain slicing from 1.120 ms to 0.098 ms; Implementing _slicechunktosize into C would yield further improvements. However, the performance seems good enough for now.

cext: increment the `parsers` module version We introduced a new `slicechunktodensity` method to the index.

sparse-revlog: introduce native (C) implementation of slicechunktodensity This is a C implementation of `_slicechunktodensity` in the `mercurial/revlogutils/deltas.py` file. The algorithm involves a lot of integer manipulation and low-level access to index data. Having a C implementation of it raises a large performance improvement. See later changeset in this series for details.

sparse-revlog: add a `trim_endidx` function in C We are about to implement a native version of `slicechunktodensity`. For clarity, we introduce the helper functions first. This function implement a subpart of the python function `_trimchunk` in `mercurial/revlogutils/deltas.py`. Handling of actual Python objects is left to the caller function.

sparse-revlog: add a `index_segment_span` function in C We are about to implement a native version of `slicechunktodensity`. For clarity, we introduce the helper functions first. This new function provides an efficient way to retrieve some of the information needed by `slicechunktodensity`.

sparse-revlog: add a `index_get_length` function in C We are about to implement a native version of `slicechunktodensity`. For clarity, we introduce the helper functions first. This new function provides an efficient way to retrieve some of the information needed by `slicechunktodensity`.

sparse-revlog: add a `index_get_start` function in C We are about to implement a native version of `slicechunktodensity`. For clarity, we introduce the helper functions first. This new function provides an efficient way to retrieve some of the information needed by `slicechunktodensity`.

perf: add --clear-revlog flag to branchmapload Having the changelog index already loaded when loading the branchmap can have a large impact on performance. Example runs (large private repository): hg perfbranchmapload -f base ! wall 0.116722 comb 0.120000 user 0.110000 sys 0.010000 (best of 59) hg perfbranchmapload -f base --clear-revlogs ! wall 0.258246 comb 0.230000 user 0.220000 sys 0.010000 (best of 31)

perf: introduce a function to fully "unload" a changelog The function remove various attributes and caches related to changelog. This is getting a common requirement.

perf: use an explicit function in perfbranchmapload This make things clearer.

perf: update function name to match `perfbranchmapload` command Having function with the same name as the command is simpler.

perf: fix a minor typo in perfbranchmapload

context: floor adjustlinkrev graph walk during copy tracing The `_adjustlinkrev` method gains an optional "stoprev" argument. The linkrev adjustment will give up once this floor is reached. The relevant functions using `_adjustlinkrev` are updated to pass an appropriate value in the copy tracing code. In some private repository, about 10% of the status call triggered the pathological case addressed by this change. The speedup varies from one call to another, the best-observed win is moving from 170s to 11s. The effect of this change can be seen in the public pypy repository, running the following command: hg perftracecopies --source 83c9ff0c0206 --destination 59c79103d5b0 before: 3.401753 seconds after: 2.634897 seconds (-23%)

context: small refactoring of `isintroducedafter` This make the next change clearer.

context: split `introrev` logic in a sub function We want to add a mechanism to stop iteration early associated to intro rev early in some case. However, it does not make sense to expose it in the public `filectx` API. So we split the code into an internal method instead.

context: introduce an `isintroducedafter` method and use it in copies Right now, copy tracing make effort to not traverse the graph too much to save performance. It uses a "limit" acting as a floor revision past which data are no longer relevant to the current copy tracing. However, to enforce this limit, it does a call to `filectx.rev()` and that call can trigger a graph traversal on its own. That extra graph traversal is unaware of the current limit and can become very expensive. That cost is increased by the nature of work done in adjust link rev, we are not only walking down the graph, we are also checking the affected file for each revision we walk through. Something significantly more expensive than the walk itself. To work around this we need to make the `filectx` operation aware of the current limit. The first step is to introduce a dedicated method: `isintroducedafter`. We'll then rework that method logic to stop traversal as soon as possible.

context: take advantage of `_descendantrev` in introrev if available Before this changeset, `_descendantrev` was ignored and `introrev` could return a "wrong" result. I was previously fine because there seems to be no existing code using both `introrev` and `_descendantrev` at the same time. However, we would like to change that.

context: spell out the logic around linkrev adjustement starting point We make the intent of the `_changeid` and `_changectx` checks explicit. The same logic was previously performed by the `self.rev()` call. The new code is a bit redundant, but much clearer.

perf: add a new `perfhelper-tracecopies` command The command is not measuring performance itself, it digs interesting statistic to help pick good arguments for the `perfcopytrace` command.

narrow: extract helper for parsing narrowspec file This will be used for parsing a file that's stored in a different location. Differential Revision: https://phab.mercurial-scm.org/D5277

py3: cast attribute name to sysstr in clearcachedproperty()

py3: whitelist three more passing test caught by buildbot The number of failing test is back to two digits. Differential Revision: https://phab.mercurial-scm.org/D5293

tests: make test-check-module-imports more robust It failed for me without this in this way: tests/test-commandserver.t:19: relative import of stdlib module tests/test-lfs-serve.t:108: relative import of stdlib module tests/test-lfs-serve.t:255: relative import of stdlib module tests/test-lfs-serve.t:362: relative import of stdlib module tests/test-lfs-serve.t:406: relative import of stdlib module tests/test-lock.py:9: imports not lexically sorted: silenttestrunner < unittest, True, True tests/test-lrucachedict.py:5: imports not lexically sorted: silenttestrunner < unittest, True, True tests/test-match.py:5: imports not lexically sorted: silenttestrunner < unittest, True, True tests/test-remotefilelog-datapack.py:15: imports not lexically sorted: silenttestrunner < unittest, True, True tests/test-remotefilelog-histpack.py:14: imports not lexically sorted: silenttestrunner < unittest, True, True tests/test-simplekeyvaluefile.py:4: imports not lexically sorted: silenttestrunner < unittest, True, True tests/test-sshserver.py:6: imports not lexically sorted: silenttestrunner < unittest, True, True This is because every module is considered a stdlib module, because the stdlib_prefixes is /usr, and my repo is in /usr/local/home, which means that sys.path contains a couple of /usr/local/home/../hg/.. entries that count as "in the stdlib". Fix this by preventing any path in sys.path that's inside the mercurial source from being considered "in the stdlib". Differential Revision: https://phab.mercurial-scm.org/D5294

push: add --publish flag to change phase of pushed changesets This flag is copied from topic extension, where it proved to be useful. It makes all pushed changesets public on both ends if the push succeeds. Doesn't currently work for any subrepos that are implicitly pushed together with their owner repo.

match: reformat `syntaxes` dictionary for better maintainability

match: provide and use a quick way to escape a single byte The previous function has a lot of overhead (including being a function). In the `_globre` case, we always escape a single byte. So we provide a dictionary dedicated to this use case. We directly use the dictionary to avoid a function call, these are expensive in Python. Again, this raise a very significant performance gain: Before: ! wall 0.059793 comb 0.060000 user 0.060000 sys 0.000000 (median of 100) After: ! wall 0.020390 comb 0.020000 user 0.020000 sys 0.000000 (median of 146) Total improvement for the full series: Before: ! wall 0.153153 comb 0.150000 user 0.150000 sys 0.000000 (median of 66) After: ! wall 0.020390 comb 0.020000 user 0.020000 sys 0.000000 (median of 146)

perf: make `clearfilecache` helper work with any object If the object is not a localrepo, it won't have an `unfiltered` method (and won't need one).

perf: move some of the perftags benchmark to the setup function Creating fresh objects and clearing the cache should not be part of the timing. before: ! wall 0.020851 comb 0.020000 user 0.020000 sys 0.000000 (median of 138) after: ! wall 0.018740 comb 0.020000 user 0.020000 sys 0.000000 (median of 141)

perf: use the new setup function in "perfbookmarks" This command was picked arbitrarily to display the usefulness of the new feature. In my Mercurial repository (with very few bookmarks), moving cache cleanup in the dedicated setup function has a visible and stable effect on the benchmark number. before: ! wall 0.000061 comb 0.000000 user 0.000000 sys 0.000000 (median of 40837) after: ! wall 0.000058 comb 0.000000 user 0.000000 sys 0.000000 (median of 40500)

perf: add a `setup` argument to run code outside of the timed section With this new argument, it is possible to perform special setup and cleanup outside of code actually timed. This is useful to provide more accurate benchmark.

perf: explicitly pass title as a keyword argument in `perfdiffwd` This will help to update the timer function arguments in a later changeset.

ui: unify argument name of ui.log() It's called an "event" in both blackbox and logtoprocess.

logtoprocess: extract logger class from ui wrapper It provides the same interface as the blackboxlogger. The ui wrapper will be removed shortly.

merge with stable

py3: use node.hex(m.digest()) instead of m.hexdigest() hashlib.sha1.hexdigest() returns str on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5287

tests: stabilize the recent checkexec changes on Windows This goes with bd0874977a5e.

tests: add Balto configuration file I have been developing a Mercurial test runner plugin for being able to run Mercurial tests with Balto (https://bitbucket.org/lothiraldan/balto/src/default/). Balto requires a configuration file so let's include it, I have added the required information in the configuration file as comments. I hope Balto would be an helpful tool for other people than me. Differential Revision: https://phab.mercurial-scm.org/D5283

tests: make test-alias.t pass with re2 Locally, these "non-ASCII character in alias" errors don't show up, though I get them when the alias is defined at the command line rather than in an hgrc. The brokenness comes from the fact that hgrcs are parsed with regexes, and re/re2 differ in this way: $ python -c 'import re; print(re.compile("(.*)").match("aaa\xc0bbbb").groups())' ('aaa\xc0bbbb',) $ python -c 'import re2; print(re2.compile("(.*)").match("aaa\xc0bbbb").groups())' ('aaa',) Apparently re2 stops when it encounters invalid utf8 (which I suppose makes sense given that '.' matches what appears to be a codepoint rather than a byte). This is presumably a bug in hg, but not very important, so just change the test to stick to valid utf8. Differential Revision: https://phab.mercurial-scm.org/D5288

context: remove seemingly impossible code branch I'm not a Python expert, but I can't think of a way that the following branch can ever be hit: def _changeid(self): if r'_changeid' in self.__dict__: return self._changeid It seems to me that if that condition is true, then this function would not have been called. The only exception I can think of is if a reference to the function had been stored beforehand, something like this: c = fctx.__dict__['_changeid'] fctx._changeid c() But that seems like very unlikely code to exist. The condition was added in 921b64e1f7b9 (filecontext: use 'is not None' to check for filelog existence, 2013-05-01) as a "bonus" change (in addition to what the patch was actually about) Differential Revision: https://phab.mercurial-scm.org/D5289

checkexec: create destination directory if necessary Since 460733327640, a "share" use the cache of the source repository. A side effect is that no `.hg/cache` directory exists in the "share" anymore. As a result, the checkexec logic can't use it to create its temporary file and have to use the working copy for that. This is suboptimal, it pollutes the working copy and prevents them to keep the file around in cache. We do not want to use the cache directory for the share target, it might be on a different file system. So instead, we (try to) create the directory if it is missing. This is a simple change that fixes the current behavior regression on stable. On default, we should probably ensure the proper directories are created when initializing the repository. We should also introduce a 'wcache' directory to hold cache file related to the working copy. This would clarify the cache situation regarding shares. The tests catch a couple of other affected cases.

py3: use pycompat.byteskwargs() to covert keys of kwargs to bytes Differential Revision: https://phab.mercurial-scm.org/D5286

py3: use '%d' for integers instead of '%s' Differential Revision: https://phab.mercurial-scm.org/D5285

py3: add 9 new passing tests caught by buildbot Differential Revision: https://phab.mercurial-scm.org/D5284

branch: allow changing branch of merge commits with --rev Tests show that changing branch of merge commits works fine with evolution and without, so let's allow it. Other safeguards should prevent users from shooting themselves in the foot.

lfs: ensure that the return of urlopener.open() is closed No problem observed, just an oversight noticed while reading documentation.

changegroup: avoid instantiating storage if we are not using it Differential Revision: https://phab.mercurial-scm.org/D5280

http: allow 'auth.prefix' to have a username consistent with the URI It may be a little weird to put a username in the prefix, but the documentation doesn't disallow it, and silently disallowing it has caused confusion[1]. The username must match what is passed in (which seems to be from the URI via a circuitous route), as well as 'auth.username' if it was specified. I thought about printing a warning for a mismatch, but we already don't print a warning if the 'auth.username' and URI username don't match. This change allows the first and second last new test cases to work as expected. It looks like this would have been a problem since at least 0593e8f81c71. [1] https://www.mercurial-scm.org/pipermail/mercurial/2018-November/051069.html

lfs: make the exception messages consistent I don't love that it repeats 'HTTP Error' in an already long message, but I doubt that we should assume that it will always say that on the original exception message.

lfs: handle URLErrors to add additional information Sometimes the blob server is hit first (e.g. on push), and sometimes it's hit last (e.g. pull). Throw in depth first subrepo operations, and things quickly get insane. It wasn't even mentioning LFS, so just saying "connection refused" can be confusing- especially if the blob server is a secondary server and connecting to the repo server works. The exception handler for the transfer handler will print the full path to the blob, but that seems fine given that it might be necessary to debug a second server. (We don't yet support a standalone blob server, so the handler for the Batch API will cover 99.9% of the current problems. But it might as well be handled now while I'm thinking about it.) The function for translating to a message was mostly borrowed from scmutil.catchall().

lfs: improve the hints for common errors in the Batch API The previous message was too debug-ish and less action oriented than a hint should be. The remaining errors that aren't handled are more along the lines of programming errors (not using POST, bad accept type, etc), so I'm not bothering with that. The friendly errors purposely use `self.baseurl` instead of the full Batch API endpoint because I'd expect some copy/paste/modify on the part of the user here, and it would be more confusing if '/objects/batch' magically appeared, but shouldn't be used in the config setting. It still seems like the right thing for debugging in the catchall case.

lfs: provide more Batch API error info via a hint in the raised exception A coworker had a typo in `lfs.url`, forgot it was even set because usually the blob server is inferred, and then got a 404. It would have been easier to debug with the failing URL printed.

scmutil: display the optional hint when handling StorageError in catchall() Other than CensoredNodeError (which is also a StorageError), it looks like all exceptions with a hint display them. I'm not sure that it makes sense to have a hint for censored nodes, so I'm not bothering with that. It looks like nobody is using this yet, as the tests don't change.

sparse-revlog: align endrevidx usages in the _slicechunktosize All "startrevidx..endrevidx" ranges in this function are now half-open.

graft: do not try to skip rev derived from ancestor more than once (issue6024) We check 'x in revs' in other cases, so let's do the same. The test case credits to Tom Prince.

subrepo: print the status line before creating the peer for better diagnostics I ran into a problem where I tried updating to a different branch, and the process appeared to hang. It turned out that the subrepo revision wasn't available locally, and I must have originally cloned it from an `hg serve -S` on a machine that currently wasn't serving anything. It took 2+ minutes to timeout, and didn't mention what it was connecting to even then. There are a couple of other issues in this scenario too. - The repo is dirty after the failed checkout because the top level repo is updated first. We should probably make 2 passes- top down to pull everything needed, and then do an update once everything is in place. - Something must be reading .hgsubstate from wdir because if the same merge command is run after the timeout, a prompt is issued that the local and remote subrepo diverged, instead of hanging. But it lists the local version and remote version as having the same hash.

sparse-revlog: use `span` variable as intended The variable was planned to be used in the while condition but was not used yet.

tests: stabilize test-commandserver.t on Windows It looks like new test coverage in 054d0fcba2c4, rather than a code change.

histedit: conditionalize the imports of 'fcntl' and 'termios' The recent import of chistedit in c36175456350 made Windows sad. I'm not sure if there's other stuff that needs to be done here (e.g. change the default interface), but this makes the tests run again. It would have been nicer if the error message indicated these modules were the problem, but instead it said "*** failed to import extension histedit: No module named histedit". I'm not sure if there's anything we can do about that.

logtoprocess: update commandfinish options arguments d2c997b8001f changed the logtoprocess API with the effect of not exposing the positional arguments to the logtoprocess scripts anymore. We have some scripts that use the duration and return code of the "commandfinish" event to monitor hg calls. Update the logging of the "commandfinish" to expose those values as options argument, which will be accessible as `OPT_RETURN_CODE` and `OPT_DURATION` in logtoprocess arguments. The code has been formatted with Black. Differential Revision: https://phab.mercurial-scm.org/D5282

rebase: fix two ui.logs to actually have text when using default blackbox log Some implementations of ui.log record structured information along with the ui.log which can be used for metrics, but ui.log() as implemented by the blackbox logging does not do anything special with this, and we end up with a log line with no text (not even a line break) so it ends up looking something like: date time user @node (pid) [rebase]> date time user @node (pid) ... Differential Revision: https://phab.mercurial-scm.org/D5279

wireprotov2server: let repo.narrowmatch(match) do matcher intersection This is supported since 4fd0fac48922 (localrepo: allow narrowmatch() to accept matcher to intersect with, 2018-09-28). Differential Revision: https://phab.mercurial-scm.org/D5281

blackbox: extract function to test if log event is tracked This will be a required method of the logger interface.

blackbox: initialize inlog flag properly And ditch the "bb" prefix as it's no longer a ui extension class.

blackbox: initialize repo attribute properly And ditch the "bb" prefix as it's no longer a ui extension class.

blackbox: unindent "if True" block

blackbox: extract logger class from ui wrapper This moves most functions to new blackboxlogger class. The ui wrapper will be removed later.

blackbox: rename variables to prepare extracting core logic from ui wrapper I'm going to add ui.setlogger() function so that I can enable logging feature in command server without extending ui.__class__. This prepares for it. "self" will be a logger instance, so this patch renames some of them to "ui".

sparse-revlog: rework the way we enforce chunk size limit We move from a O(N) algorithm to a O(log(N)) algorithm. The previous algorithm was traversing the whole delta chain, looking for the exact point where it became too big. This would result in most of the delta chain to be traversed. Instead, we now use a "binary" approach, slicing the chain in two until we have a chunk of the appropriate size. We still keep the previous algorithm for the snapshots part. There are few of them and they are large bits of data distant from each other. So the previous algorithm should work well in that case. To take a practical example of restoring manifest revision '59547c40bc4c' for a reference NetBeans repository (using sparse-revlog). The media time of the step `slice-sparse-chain` of `perfrevlogrevision` improve from 1.109 ms to 0.660 ms.

doctest: add a `issnapshot` method to _testrevlog We'll need it soon.

tests: add `revlogutils.deltas` module to doctests The doctest in these module have been from `mercurial.revlog` but the module was not added to the doctests. Spotted by Yuya Nishihara.

merge with stable

mergetools: adjust Beyond Compare config on Mac/Linux Set the labels of the Linux and Mac versions of Beyond Compare from Mercurial's builtin variables, same as the Windows version. Differential Revision: https://phab.mercurial-scm.org/D5255

tests: allow for 100% of profiled time in sleep in test-profile.t I'm getting an annoying failure in this test on our builder, and I *think* what's happening is that the profiler is taking _just_ long enough to start that we're spending 100% of the profiled time in the sleep function, which was causing the leading space to not be printed since the 100 was in the first column of output. Differential Revision: https://phab.mercurial-scm.org/D5272

rewriteutil: move publicrevs closer to where it's used

requires: use atomictemp=True when writing .hg/requires We use an unusual file system at Google that allows writes (and renames) but not deletions (for certain paths). That causes problems when writing the requires files without atomictemp=True. There doesn't appear to be any real drawbacks to using atomictemp, so I'm hoping we can just change it in core. Differential Revision: https://phab.mercurial-scm.org/D5274

blackbox: extract _log() function which is called after lastui is resolved This makes sure that self is the solo ui instance used in _log().

blackbox: inline temporary variables which are referenced only once

blackbox: simply update global lastui variable at once

blackbox: consolidate conditions for early return Just pick the lastui only if it is usable.

blackbox: remove redundant check for unassigned repo Since ui._bbvfs is looked through ui._bbrepo, the repo instance should exist if ui._bbvfs isn't None.

tests: fix bytes/str issue I introduced when adding this test # skip-blame just b prefixes for py3 Differential Revision: https://phab.mercurial-scm.org/D5271

shelve: use matcher to restrict prefetch to just the modified files Shelve currently operates by: - make a temp commit - identify all the bases necessary to shelve, put them in the bundle - use exportfile to export the temp commit to the bundle ('file' here means "export to this fd", not "export this file") - remove the temp commit exportfile calls prefetchfiles, and prefetchfiles uses a matcher to restrict what files it's going to prefetch; if it's not provided, it's alwaysmatcher. This means that `hg shelve` in a remotefilelog repo can possibly download the file contents of everything in the repository, even when it doesn't need to. It luckily is restricted to the narrowspec (if there is one), but this is still a lot of downloading that's just unnecessary, especially if there's a "smart" VCS-aware filesystem involved. exportfile is called with exactly one revision to emit, so we're just restricting it to prefetching the files from that revision. The base revisions having separate files should not be a concern since they're handled already; example: commit 10 is draft and modifies foo/a.txt and foo/b.txt commit 11 is draft and modifies foo/a.txt my working directory that I'm shelving modifies foo/b.txt By the time we get to exportfile, commit 10 and 11 are already handled, so the matcher only specifying foo/b.txt does not cause any problems. I verified this by doing an `hg unbundle` on the bundle that shelve produces, and getting the full contents of those commits back out, instead of just the files that were modified in the shelve. Differential Revision: https://phab.mercurial-scm.org/D5268

revlog: automatically read from opened file handles The revlog reading code commonly opens a new file handle for reading on demand. There is support for passing a file handle to revlog.revision(). But it is marked as an internal argument. When revlogs are written, we write() data as it is available. But we don't flush() data until all revisions are written. Putting these two traits together, it is possible for an in-process revlog reader during active writes to trigger the opening of a new file handle on a file with unflushed writes. The reader won't have access to all "available" revlog data (as it hasn't been flushed). And with the introduction of the previous patch, this can lead to the revlog raising an error due to a partial read. I witnessed this behavior when applying changegroup data (via `hg pull`) before issue6006 was fixed via different means. Having this and the previous patch in play would have helped cause errors earlier rather than manifesting as hash verification failures. While this has been a long-standing issue, I believe the relatively new delta computation code has tickled it into being more common. This is because the new delta computation code will compute deltas in more scenarios. This can lead to revlog reading. While the delta computation code is probably supposed to reuse file handles, it appears it isn't doing so in all circumstances. But the issue runs deeper than that. Theoretically, any code can access revision data during revlog writes. It appears we were just getting lucky that it wasn't. (The "add revision callback" passed to addgroup() provides an avenue to do this.) If I changed the revlog's behavior to not cache the full revision text or to clear caches after revision insertion during addgroup(), I was able to produce crashes 100% of the time when writing changelog revisions. This is because changelog's add revision callback attempts to resolve the revision data to access the changed files list. And without the revision's fulltext being cached, we performed a revlog read, which required opening a new file handle. This attempted to read unflushed data, leading to a partial read and a crash. This commit teaches the revlog to store the file handles used for writing multiple revisions during addgroup(). It also teaches the code for resolving a file handle when reading to use these handles, if available. This ensures that *any* reads (regardless of their source) use the active writing file handles, if available. These file handles have access to the unflushed data because they wrote it. This allows reads to complete without issue. Differential Revision: https://phab.mercurial-scm.org/D5267

revlog: detect incomplete revlog reads _readsegment() is supposed to return N bytes of revlog revision data starting at a file offset. Surprisingly, its behavior before this patch never verified that it actually read and returned N bytes! Instead, it would perform the read(), then return whatever data was available. And even more surprisingly, nothing in the call chain appears to have been validating that it received all the data it was expecting. This behavior could lead to partial or incomplete revision chunks being operated on. This could result in e.g. cached deltas being applied against incomplete base revisions. The delta application process would happily perform this operation. Only hash verification would detect the corruption and save us. This commit changes the behavior of raw revlog reading to validate that we actually read() the number of bytes that were requested. We will raise a more specific error faster, rather than possibly have it go undetected or manifest later in the call stack, at delta application or hash verification. Differential Revision: https://phab.mercurial-scm.org/D5266

revlog: use single file handle when de-inlining revlog _getsegmentforrevs() will eventually call into _datareadfp() to resolve a file handle to read revision data. If no file handle is passed into _getsegmentforrevs(), it opens a new one. Explicit is better than implicit. This commit changes _enforceinlinesize() to open a file handle explicitly when converting inline revlogs to split revlogs and to pass this file handle into _getsegmentforrevs(). I haven't measured, but this change should improve performance, as we no longer reopen the revlog for reading for every revision in the revlog when it is converted from inline to split. Instead, we open it at most once and use it for the duration of the operation. That being said, I /think/ the chunk cache may mitigate the number of file opens required. Differential Revision: https://phab.mercurial-scm.org/D5265

store: raise ProgrammingError if unable to decode a storage path Right now, the function magically return False which is dangerous, so let's raise ProgrammingError. Suggested by Augie in D5139. Differential Revision: https://phab.mercurial-scm.org/D5264

tests: document a known failing interaction between narrow and lfs This is one of the two remaining aborts I found looking into issue5794. I've got no idea what's wrong with the hook, since the changes there fixed the other two problems noted in that bug report. It seems like it might go away when the narrow issue is fixed, but let's make sure this doesn't get lost. The stacktrace for the hook seems to indicate that the missing file *is* in ctx: remote: Traceback (most recent call last): remote: File "c:\Users\Matt\projects\hg\hgext\lfs\__init__.py", line 253, in checkrequireslfs remote: if any(f in ctx and match(f) and ctx[f].islfs() for f in ctx.files()): remote: File "c:\Users\Matt\projects\hg\hgext\lfs\__init__.py", line 253, in <genexpr> remote: if any(f in ctx and match(f) and ctx[f].islfs() for f in ctx.files()): remote: File "c:\Users\Matt\projects\hg\hgext\lfs\wrapper.py", line 191, in filectxislfs remote: return _islfs(self.filelog(), self.filenode()) remote: File "c:\Users\Matt\projects\hg\mercurial\context.py", line 631, in filenode remote: return self._filenode remote: File "c:\Users\Matt\projects\hg\mercurial\util.py", line 1528, in __get__ remote: result = self.func(obj) remote: File "c:\Users\Matt\projects\hg\mercurial\context.py", line 579, in _filenode remote: return self._filelog.lookup(self._fileid) remote: File "c:\Users\Matt\projects\hg\mercurial\filelog.py", line 68, in lookup remote: self._revlog.indexfile) remote: File "c:\Users\Matt\projects\hg\mercurial\utils\storageutil.py", line 218, in fileidlookup remote: raise error.LookupError(fileid, identifier, _('no match found')) remote: LookupError: data/inside2/f.i@f59b4e021835: no match found

logtoprocess: drop support for ui.log() call with invalid msg arguments (BC) Before, the logtoprocess extension put a formatted message into $MSG1, and its arguments to $MSG2... If the specified arguments couldn't be formatted because of a caller bug, an unformatted message was passed in to $MSG1 instead of exploding. This behavior doesn't make sense. Since I'm planning to formalize the ui.log() interface such that we'll no longer have to extend the ui class, I want to remove any features not conforming to the ui.log() API. So this patch removes the support for ill-formed arguments, and $MSG{n} (where n > 1) parameters which seems useless as long as the message can be formatted. The $MSG1 variable isn't renamed for the maximum compatibility. In future patches, a formatted msg will be passed to a processlogger object, instead of overriding the ui.log() function. .. bc:: The logtoprocess extension no longer supports invalid ``ui.log()`` arguments. A log message is always formatted and passed in to the ``$MSG1`` environment variable.

py3: byte-stringify inline extension in test-logtoprocess.t

logtoprocess: rewrite dict building in py3-compatible way

logtoprocess: leverage procutil.shellenviron() to stringify variables (BC) This should make the extension more Py3 friendly. The environment variables of the main process are copied to the dict by shellenviron(). .. bc:: Boolean options passed to the logtoprocess extension are now formatted as ``0`` or ``1`` instead of ``None``, ``False``, or ``True``.

templater: check invalid use of list expression properly (issue5920) The error message is still cryptic, but it should be better.

copystore: provide unit to ui.makeprogress()

verify: provide unit to ui.makeprogress()

tests: fix wireproto redirection test on systems without tls1.2 Our automated package builder has some ancient configuration that lacks modern TLS, which is how we noticed this. Tested: the test now passes on both macOS High Sierra (has tls1.2) and Ubuntu Trusty (which does not).

py3: add b suffix to make sure file is opened in bytes mode Differential Revision: https://phab.mercurial-scm.org/D5263

py3: don't use dict.iterkeys() dict.iterkeys() is not present on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5262

py3: use node.hex(h.digest()) instead of h.hexdigest() hashlib.sha1.hexdigest() returns str on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5261

py3: make second argument of fdopen() a str # skip-blame as just r'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5260

py3: fix keyword arguments handling in hgext/remotefilelog/ Keys of kwargs on Python 3 should be strings. This patch fixes them by appending r'' prefixes, and using pycompat.byteskwargs() and pycompat.strkwargs(). Differential Revision: https://phab.mercurial-scm.org/D5259

py3: add __pycache__ to .hgignore Something changed with Python 3 just like million on things. Maybe they should named Python 3 as anaconda or cobra instead of just increasing the version number. This makes test-custom-filters.t pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5039

sparse-revlog: stop using a heap to track selected gap Same logic as for 'gapsheap', we don't actually need a heap.

sparse-revlog: stop using a heap to track gaps The heap doesn't bring any performance advantage as we can simply sort the final list. Moreover, the lesser complexity helps a lot when we later implement it in C.

sparse-revlog: fast-path before computing payload size In this fast-path case, we do not need to compute the full delta chain payload. Since it comes with a significant cost, we avoid doing so if possible.

sparse-revlog: drop unused deltainfo parameter from segmentspan We no longer need to deal with slicing logic including uncommitted revision. We drop the associated code.