tests: remove shell function helper from test-largefiles-misc Now that all the complexity is in a Python script, we can just directly invoke the tool. Differential Revision: https://phab.mercurial-scm.org/D1599

contrib: ban find(1)'s -printf operator, as it is a GNU-ism Differential Revision: https://phab.mercurial-scm.org/D1598

merge with stable

repoview: include filter name in repr for debugging

repoview: extract a factory function of proxy class This makes sure that dynamically-created class objects are isolated from local binding of repo instances. The type cache is moved to module level as it isn't tied to each instance.

repoview: do not include filter name in name of proxy class The type object is shared across all filters. I'll add __repr__() instead.

setup: convert version strings to unicode on Python 3 Fixes the following error: stderr from 'hg log -T x -r only(.,'b'4.4.2'')': b' hg: parse error at 10: unexpected token: symbol'

thirdparty: move selectors2 module to where it should be

rewriteutil: use precheck() in uncommit and amend commands Differential Revision: https://phab.mercurial-scm.org/D1526

rewriteutil: add a precheck function to check if revs can be rewritten The precheck function is intended to be used before we start rewritting changesets. Differential Revision: https://phab.mercurial-scm.org/D1503

rewriteutil: add utility function to check if we can create new unstable cset This patch adds a new file which will contain utility functions related to rewritting changesets. It also adds a utility function to check if the rewritting operation creates new unstable changesets and are we allowed to create them. This rewriteutil.py introduced in this patch and the utility functions added in the upcoming patches exists in the evolve extension are being ported from there. Differential Revision: https://phab.mercurial-scm.org/D1502

test-run-tests: do not rebuild hg in the test d600bda4 and fc0f3ed0 added code to call `$PYTHON run-tests.py ...`. That will rebuild hg, is slow and could have other crashes from setup.py, like: Unable to find a working hg binary to extract the version from the repository tags Therefore use `run-tests.py -l` instead. Differential Revision: https://phab.mercurial-scm.org/D1595

remotenames: consider existing data while storing newer data Previously reviewed as D1357. Differential Revision: https://phab.mercurial-scm.org/D1551

remotenames: add functions to read remotenames data from .hg/remotenames/ This patch functions which can be used to read remotenames data from .hg/remotenames/. The logic for the function which reads the remotenames file is taken from the remotenames extension. Previously reviewed as D940. Differential Revision: https://phab.mercurial-scm.org/D1550

remotenames: add test showing overwriting on remotenames data The current storage logic every time overwrites the existing data with the new data. This patch adds test to demonstrate that. To fix this, we need to add logic to read existing remotenames data and merge with existing data which will be added in upcoming changesets. Previously reviewed as D1356. Differential Revision: https://phab.mercurial-scm.org/D1549

remotenames: add functionality to store remotenames under .hg/hgremotenames/ This patch moves the functionality from remotenames extension to store remotenames to core. Storage format used by remotenames extension: A single file `.hg/remotenames` with an entry in each line where each line is of format: `node nametype remotepath/name` where nametype is either 'bookmarks' or 'branches'. This was not the best way to store data, so while moving to core the storage format was changed but yet not the final format. The storage format used by core after this patch will be: * A file for each type of name i.e. bookmarks and branches in .hg/remotenames/ directory * A version number on the top of the file. The version for current format is 0. * An entry in each line where each line is of the format `node\0remotepath\0name` The logic to sync with existing remotenames file and saving journals and other related things will be moved to core in next patches incrementally. Thanks to Ryan, Augie and Durham for suggestions on storage format. Previously reviewed as D939. Differential Revision: https://phab.mercurial-scm.org/D1548

remotenames: move function to pull remotenames from the remoterepo to core This patch is the first patch of the series moving functionality from hgremotenames extension to core. There are lot of functionality in the extension which in the end enables us to store branch heads and bookmarks location on a server from which we are pulling or cloning from. This will help us in creating a better bookmark workflow where we can show user that a certain server has this bookmarks at this node. It will also introduce namespaces related to remote bookmarks and remote branches. This patch moves the functionality to pull branches and bookmarks from a server from which we are pulling to core behind config option `experimental.remotenames`. This patch adds a test which helps us to analyse whether things are working or not. We are currently writing things to ui, we will write information to files in upcoming patches. Previously reviewed as D937. Differential Revision: https://phab.mercurial-scm.org/D1547

test: fix bad replace for fixing pure-only build When we replaced the patterns, glob was removed on the fixed line, it was a mistake and caused the pure-only build to fails. Differential Revision: https://phab.mercurial-scm.org/D1592

test: fix common-pattern for pure variant The $USUAL_COMPRESSIONS$ value that was taken was not compatible with the pure variant systems as zlib seems to not be available in these case. Differential Revision: https://phab.mercurial-scm.org/D1562

tests: add a substitution for EADDRINUSE/WSAEADDRINUSE messages I suspect some more of these are globbed out, so this is a bit of future proofing.

tests: add a substitution for ECONNRESET/WSAECONNRESET messages

tests: add a substitution for ENOTDIR/ERROR_PATH_NOT_FOUND messages

tests: add a substitution for ENOENT/ERROR_FILE_NOT_FOUND messages Automatic replacement seems better than trying to figure out a check-code rule. I didn't bother looking to see why the error message and file name is reversed in the annotate and histedit tests, based on Windows or not. I originally had this as a list of tuples, conditional on the platform. But there are a couple of 'No such file or directory' messages emitted by Mercurial itself, so unconditional is required for stability. There are also several variants of what I assume is 'connection refused' and 'unknown host' in test-clone.t and test-clonebundles.t for Docker, FreeBSD jails, etc. Yes, these are handled by (re) tags, but maybe it would be better to capture those strings in order to avoid whack-a-mole in future tests. All of this points to using a dictionary containing one or more strings-to-be-replaced values.

setup: only write some autogenerated files if they change Without this change, setup.py always writes some files on every invocation. This prevents some builds from being a no-op when they should. And, since times can sneak into generated .pyc files, this prevents file content from being deterministic between builds. As part of the refactor, we treat file content as bytes. The only potential regression from this would be if some tool is looking at mtimes of the changed files to determine if further action should be taken. But I don't think anything critically important is keyed off the mtimes of these specific files. Differential Revision: https://phab.mercurial-scm.org/D1580

python3: whitelist many more passing tests Differential Revision: https://phab.mercurial-scm.org/D1584

fancyopts: fix handling of "--" value in earlygetopt()

fancyopts: use getopt.gnu_getopt() The issue described in the docstring has been fixed since Python 20ab2260dc93, which is in 2.7. https://hg.python.org/cpython/rev/20ab2260dc93 https://bugs.python.org/issue4458 This fixes the handling of '--' value.

dispatch: replace _earlygetopt(strip=True) with new parser The execution order in cmdalias.__init__() is adjusted to set stripped args to self.givenargs, which is no longer updated in place.

dispatch: replace _earlyreq*() with new fancyopts-based parser

dispatch: alias --repo to --repository while parsing early options This prepares for replacing old _early*opt() functions. My initial attempt was to extend options table to support 'repository|repo' syntax. It worked, but seemed too invasive. So I decided to add an optional argument to fancyopts() instead. This also changes the nevernegate dict to be keyed by a canonical_name, not by an option-name for clarity.

spartan: render changesets server-side on /graph page

monoblue: render changesets server-side on /graph page

gitweb: render changesets server-side on /graph page

paper: render changesets server-side on /graph page

hgweb: only include graph-related data in jsdata variable on /graph pages (BC) Historically, client-side graph code was not only rendering the graph itself, but it was also adding all of the changeset information to the page as well. It meant that JavaScript code needed to construct valid HTML as a string (although proper escaping was done server-side). It wasn't too clunky, even though it meant that a lot of server-side things were duplicated client-side for no good reason, but the worst thing about it was the data format it used. It was somewhat future-proof, but not human-friendly, because it was just a tuple: it was possible to append things to it (as was done in e.g. 270f57d35525), but you'd then have to remember the indices and reading the resulting JS code wasn't easy, because cur[8] is not descriptive at all. So what would need to happen for graph to have more features, such as more changeset information or a different vertex style (branch-closing, obsolete)? First you'd need to take some property, process it (e.g. escape and pass through templatefilters function, and mind the encoding too), append it to jsdata and remember its index, then go add nearly identical JavaScript code to 4 different hgweb themes that use jsdata to render HTML, and finally try and forget how brittle it all felt. Oh yeah, and the indices go to double digits if we add 2 more items, say phase and obsolescence, and there are more to come. Rendering vertex in a different style would need another property (say, character "o", "_", or "x"), except if you want to be backwards-compatible, it would need to go after tags and bookmarks, and that just doesn't feel right. So here I'm trying to fix both the duplication of code and the data format: - changesets will be rendered by hgweb templates the same way as changelog and other such pages, so jsdata won't need any information that's not needed for rendering the graph itself - jsdata will be a dict, or an Object in JS, which is a lot nicer to humans and is a lot more future-proof in the long run, because it doesn't use numeric indices What about hgweb themes? Obviously, this will break all hgweb themes that render graph in JavaScript, including 3rd-party custom ones. But this will also reduce the size of client-side code and make it more uniform, so that it can be shared across hgweb themes, further reducing its size. The next few patches demonstrate that it's not hard to adapt a theme to these changes. And in a later series, I'm planning to move duplicate JS code from */graph.tmpl to mercurial.js and leave only 4 lines of code embedded in those <script> elements, and even that would be just to allow redefining graph.vertex function. So adapting a custom 3rd-party theme to these changes would mean: - creating or copying graphnode.tmpl and adding it to the map file (if a theme doesn't already use __base__) - modifying one line in graph.tmpl and simply removing the bigger part of JavaScript code from there Making these changes in this patch and not updating every hgweb theme that uses jsdata at the same time is a bit of a cheat to make this series more manageable: /graph pages that use jsdata are broken by this patch, but since there are no tests that would detect this, bisect works fine; and themes are updated separately, in the next 4 patches of this series to ease reviewing.

tests: write and use a custom helper script to avoid find's -printf -printf on find is a GNU-ism and will be banned in an upcoming check-code change. Differential Revision: https://phab.mercurial-scm.org/D1597

hgweb: adopt child nodes in ajaxScrollInit on /graph pages too ajaxScrollInit is a function that loads more elements (e.g. changelog entries) when browser window is scrolled down to the bottom of the page. It basically fetches the next page from the server as HTML, finds container element in that document and "adopts" (essentially, moves) all its child nodes to the container in the current document. Currently, hgweb doesn't render any changesets on /graph page (everything is done in JavaScript), so there are no children to adopt. But there will be soon, so let's create a reusable function that does it. Hardcoding #graphnodes selector is suboptimal, but graph code already does this in two other places.

lfs: enable the extension locally after converting to an 'lfs' repo This is consistent with clone and share in the previous commits.

lfs: enable the extension locally after sharing a repo with 'lfs' requirement This is consistent with clone in the previous commit.

lfs: enable the extension locally after cloning a repo with 'lfs' requirement We do the same thing on clone for the largefiles extension, as a convenience. Similar to largefiles, it's probably safer to only enable this extension on a per repo basis because it is trivial to add an lfs file. And that gives the repository some centralized VCS characteristics.

log: translate column labels at once (issue5750) This makes sure that all columns are aligned. getlogcolumns() is hosted by templatekw so the namespaces module can see it. i18n/de.po is updated so test-log.t passes with no error. "obsolete:" and "instability:" are kept untranslated.

log: remove temporary variable 'date' used only once

merge with stable

lock: allow to configure when the lock messages are displayed We add a new 'ui.timeout.warn' config to set a grace period before we display lock related warning: waiting for lock on PATH held by PROCESS The config is based on 'ui.timeout' and expresses a number of seconds before the warning is displayed. Negative values disable the warning altogether. The messages go to the debug output to help people trouble-shooting deadlocks.

lock: add a trylock method handling the timeout and messaging logic We are about to make the messages around lock more flexible. We move all the currently logic into a function in the lock module. We'll update the message scheme in the next changeset.

lock: use configint for 'ui.timeout' config The ui object can do the conversion itself.

scmutil: improve format pattern used in nodesummaries As spotted by Yuya Nishihara, that value is an integer.

tests: trivial fixes for Windows

py3: use encoding.strtolocal() to convert string to bytes Differential Revision: https://phab.mercurial-scm.org/D1557

py3: use pycompat.bytestr() or '%d' in place of str() Differential Revision: https://phab.mercurial-scm.org/D1556

archive: pass thru mtime for directory archives, like other archive types do Without this files in the output archive directory have their mtimes set to whatever time they were written their. This is in this inconsistent with the other archivers, eg. zip, which use exactly the same time for all files. Works on my machine (Windows), but I don't have a *nix box available to run anything more thorough, unfortunately.

hgweb: remove negative top from .info line in graph "top: -Xpx" shifts a block up by X pixels, which can be used to visually compress two lines of text to have less space between them, in this case it's used for the changesets on /graph page. But not only it's not needed there (both lines fit fine into their allowed vertical space), but it would also look better (not as crammed, more vertically centered) without these negative values. "position: relative" is needed solely for the "top" property to have effect on the element, no children of the .info element rely on it, so let's remove it as well.

unamend: drop unused vars, query after taking lock, use ctx.hex() for extras This is the followup of review on D821. Differential Revision: https://phab.mercurial-scm.org/D1579

test-lfs: drop a hack for ignoring convert devel-warnings This was fixed on stable in 281214150561.

convert: restore the ability to use bzr < 2.6.0 (issue5733) This effectively conditionalizes a234b32b744a. Some Linux distributions (like CentOS 7) use really old versions, and the change referenced was causing exceptions to be thrown. Even though the deprecation warning says 'since 2.5.0', it wasn't marked as such in 2.5.1, but is by 2.6.0. This was tested with 2.4.2 and 2.6.0 with PYTHONWARNINGS=::DeprecationWarning, and both paths were exercized.

py3: use bytes in place of basestring All strings in mercurial must be bytes so we can easily replace basestring with bytes. Differential Revision: https://phab.mercurial-scm.org/D1555

amend: make a copy of "extra" to avoid mutating an input I don't know of any problems this has caused, it just seems less surprising. Differential Revision: https://phab.mercurial-scm.org/D1575

tests: removes bashism from test-unamend.t FreeBSD builders fail because of bashism. Differential Revision: https://phab.mercurial-scm.org/D1578

run-tests: avoid calculating _testdir again Differential Revision: https://phab.mercurial-scm.org/D1574

run-tests: simplify by using dict.pop() with default Differential Revision: https://phab.mercurial-scm.org/D1573

py3: use byteskwargs in sparse.py This removes several dozen failures in Python 3. Differential Revision: https://phab.mercurial-scm.org/D1482

py3: define __next__ in patch.py This needed to appease Python 3's iterator protocol. This is crasher #5 in Python 3. Differential Revision: https://phab.mercurial-scm.org/D1480

run-tests: mechanism to report exceptions during test execution Sometimes when running tests you introduce a ton of exceptions. The most extreme example of this is running Mercurial with Python 3, which currently spews thousands of exceptions when running the test harness. This commit adds an opt-in feature to run-tests.py to aggregate exceptions encountered by `hg` when running tests. When --exceptions is used, the test harness enables the "logexceptions" extension in the test environment. This extension wraps the Mercurial function to handle exceptions and writes information about the exception to a random filename in a directory defined by the test harness via an environment variable. At the end of the test harness, these files are parsed, aggregated, and a list of all unique Mercurial frames triggering exceptions is printed in order of frequency. This feature is intended to aid Python 3 development. I've only really tested it on Python 3. There is no shortage of improvements that could be made. e.g. we could write a separate file containing the exception report - maybe even an HTML report. We also don't capture which tests demonstrate the exceptions, so there's no turnkey way to test whether a code change made an exception disappear. Perfect is the enemy of good. I think the current patch is useful enough to land. Whoever uses it can send patches to imprve its usefulness. Differential Revision: https://phab.mercurial-scm.org/D1477

run-tests: make --extra-config-opt work with Python 3 And add test coverage to ensure it works. Differential Revision: https://phab.mercurial-scm.org/D1476

run-tests: organize options into argument groups And sort arguments so help output is more legible. There are probably a ton of ways to group things. I tried to picture the test harness as a pipeline and attempted to draw boundaries around stages in that pipeline to create the groupings. Differential Revision: https://phab.mercurial-scm.org/D1475

run-tests: convert to argparse optparse has been deprecated since Python 3.2. Best to get on the new boat before the old one sinks. It looks like argparse formats its usage string differently than optparse. Meh. Differential Revision: https://phab.mercurial-scm.org/D1474

merge with stable

server: introduce a 'experimental.single-head-per-branch' option When the option is set, the repository will reject any transaction adding multiple heads to the same named branch. For now we reject all scenario with multiple heads. One could imagine handling closed branches differently. We prefer to keep things simple for now. The feature might get extended later. Branch closing is not the best experience Mercurial has to offer anyway.

scmutil: extra utility to display a reasonable amount of nodes Push have some logic to display a reasonable amount nodes. We extract it to an utility function to make it reusable.

uncommit: unify functions _uncommitdirstate and _unamenddirstate to one The _unamenddirstate() function was inspired by _uncommitdirstate() function as the logic was same but we were unable to use the latter function directly. So previous patch introduced the _unamenddirstate() function and now this patch unifies both the function and we have a _fixdirstate() function. Adding function in previous patch and unifying in a later patch makes the reasoning easier and also leaves the last patch dedicated to what it is meant to be. Differential Revision: https://phab.mercurial-scm.org/D971

unamend: move fb extension unamend to core unamend extension adds an unamend command which undoes the effect of the amend command. This patch moves the unamend command from that extension to uncommit extension and this one does not completely undoes the effect of amend command as it creates a new commit, rather than reviving the old one back. This also adds tests for the same. .. feature:: A new unamend command in uncommit extension which undoes the effect of the amend command by creating a new changeset which was there before amend and moving the changes that were amended to the working directory. Differential Revision: https://phab.mercurial-scm.org/D821

color: respect HGPLAINEXCEPT=color to allow colors while scripting (issue5749) I'd also like --color=always on the command-line to override HGPLAIN=1 et al, but that's more work, and this seems like a better fix. We've got a fair number of programs that actually want to automate hg and get colored output to users, so they should set HGPLAINEXCEPT=alias (what we usually recommend), but this has been breaking them because they then lose color. .. feature:: The ``HGPLAINEXCEPT`` environment variable can now include ``color`` to allow automatic output colorization in otherwise automated environments. Differential Revision: https://phab.mercurial-scm.org/D1532

test-ssh: do not actually look up "brokenrepository" by DNS

largefiles: explicitly set the source and sink types to 'hg' for lfconvert I stumbled into this prior to adding the type indicator on the source and sink, but there's no reason to try to infer the types for this conversion.

lfs: add a repo requirement for this extension when converting to lfs This covers both the vanilla repo -> lfs repo and largefiles -> lfs conversions. The largefiles extension adds the requirement directly, because it has a dedicated command to convert. Using the convert extension is better, because it supports more features. I'd like ideas about how to ensure that converting away from lfs works on all files. (See comments in test-lfs.t)

convert: allow the sink object to be wrapped when the extension isn't loaded The next patch will wrap the conversion code, in order to write out a requirement for 'lfs' when appropriate. Wrapping convcmd.convertsink() in an afterloaded callback works fine when the convert extension is enabled by the user. The problem here is that lfconvert uses the convert extension, whether or not it was formally enabled by the user. My first attempt was to have lfs install an afterloaded callback that would wrap the convert sink if convert was loaded, or wrap lfconvert if it wasn't. Then the lfconvert override could install an afterloaded callback to try wrapping the convert sink again, before calling the original lfconvert. But that breaks down if largefiles can't load the convert extension on the fly. [1] Further, some tests were failing with an error indicating that the size of the afterloaded list changed while iterating it. Yuya mentioned that maybe some bits of convert could be moved into core, but I'm not sure where to draw that line. The convertsink() method depends on the list of sinks, which in turn depends on the sink classes. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-November/108038.html

convert: save an indicator of the repo type for sources and sinks This seems like basic info to have, and will be used shortly when deciding whether or not to wrap the class for lfs conversions. The other option is to just add a function to each class. But this seems better in that the strings aren't duplicated, and the constructor for most of these will run even if the VCS isn't installed, so it's easier to catch errors.

lfs: add a repo requirement for this extension once an lfs file is committed Largefiles does the same thing (also delayed until the first largefile commit), to prevent access to the repo without the extension. In the case of this extension, not having the extension loaded while accessing an lfs file results in cryptic errors about "missing processor for flag '0x2000'". If enabled locally but not remotely, the cryptic error message is about no common changegroup version. (It wants '03', which is currently experimental.) The largefiles extension looks for any tracked file that starts with '.hglf/'. Unfortunately, that doesn't work here. I didn't see any way to get the files that were just committed, without doing a full status. But since there's no secondary check on adding an lfs file once the extension is loaded and a threshold set, the best practice is to only enable this locally on a repo that needs it. That should minimize the unnecessary overhead for repos without an lfs file.

Added signature for changeset a92b9f8e11ba

Added tag 4.4.2 for changeset a92b9f8e11ba

merge: check created file dirs for path conflicts only once (issue5716) In large repositories, updates involving the creation of many files check the same directories repeatedly in the wctx manifest. Move these checks out to a separate loop to avoid repeated checks hitting the manifest. Differential Revision: https://phab.mercurial-scm.org/D1226

merge: cache unknown dir checks (issue5716) As mentioned in D1222, the recent pathconflicts change regresses update performance in large repositories when many files are being updated. To mitigate this, we introduce two caches of directories that have already found to be either: - unknown directories, but which are not aliased by files and so don't need to be checked if they are files again; and - missing directores, which cannot cause path conflicts, and cannot contain a file that causes a path conflict. When checking the paths of a file, testing against this caches means we can skip tests that involve touching the filesystem. Differential Revision: https://phab.mercurial-scm.org/D1224

dispatch: add HGPLAIN=+strictflags to restrict early parsing of global options If this feature is enabled, early options are parsed using the global options table. As the parser stops processing options when non/unknown option is encountered, it won't mistakenly take an option value as a new early option. Still "--" can be injected to terminate the parsing (e.g. "hg -R -- log"), I think it's unlikely to lead to an RCE. To minimize a risk of this change, new fancyopts.earlygetopt() path is enabled only when +strictflags is set. Also the strict parser doesn't support '--repo', a short for '--repository' yet. This limitation will be removed later. As this feature is backward incompatible, I decided to add a new opt-in mechanism to HGPLAIN. I'm not pretty sure if this is the right choice, but I'm thinking of adding +feature/-feature syntax to HGPLAIN. Alternatively, we could add a new environment variable. Any bikeshedding is welcome. Note that HGPLAIN=+strictflags doesn't work correctly in chg session since command arguments are pre-processed in C. This wouldn't be easily fixed.

fancyopts: add early-options parser compatible with getopt() The next patch will add a flag for strict parsing of early options, where we'll have to parse all early options at once instead of processing them one-by-one by dispatch._earlygetopt(). That's why I decided to hook fancyopts(). All dispatch._early*opt() functions is planned to be replaced with this function. But in this stable series, only the strict mode will be handled by fancyopts.earlygetopt().

largefiles: pay attention to dropped standin files when updating largefiles Previously, the largefile for a dropped standin would be deleted here, and then restored from the cache. This had the effect of clobbering uncommitted changes if a revert caused the file to be forgotten, which is not what happens with a normal file. Now the removal and update is skipped for dropped largefiles, and the corresponding standin is deleted from disk. This was noticed when working on issue5738 because the forgotten standin files were left behind, and that changes the behavior of the next rename to that directory. My first attempt was to cleanup the standins before calling this. That failed, because this function deletes the largefile if the corresponding standin is missing. This function is called by the revert command, merge (and therefore update), and patch, via the scmutil.marktouched() override. So it should be pretty narrow in scope. I didn't mark issue5738 as fixed because the move related issues can still happen if the main tree and the .hglf subtree get out of sync somehow. I don't see an easy fix for that, but that should be an edge case. If whoever queues this thinks it is good enough to close out the bug and can cram it into the summary, go for it.

test-largefiles: demonstrate problems with renaming and reverting a directory These things were uncovered looking at issue5738. First, if the destination directory exists under .hglf, the source is moved under the destination instead of renaming the last component for `hg mv srcdir dstdir`. This is extra confusing, because it occurs even if the user visible destination (i.e. the path _not_ under .hglf) does not exist. Additionally, when a largefile is forgotten via revert, any modifications end up getting clobbered. For normal files, the forgotten file is left unchanged, as shown by test-import.t. The forget command on a largefile will correctly leave the file unmodified.

tests: add globs for Windows

cat: record the current behavior of wildcard matches in subrepos Mercurial subrepos support any match patterns.

match: remove doc about undefined behavior of visitdir() This was added by 8545bd381504, but core matchers support visitdir() of arbitrary locations since 2773540c3650, and verifier._verifymanifest() doesn't seem to strictly obey the restriction. I have no idea how important this API contract is for third-party extensions. That's why this patch is RFC.

merge with stable

hgweb: add .jshintrc with some basic rules This file is picked up automatically by jshint, so no extra changes required in test-check-jshint.t.

hgweb: look up "URLSearchParams" in "window" to work around jshint issues Unfortunately, current version of jshint (2.9.5) doesn't know such a global variable and complains that it's undefined. Since this line tries to look up URLSearchParams in a global scope (i.e. window), let's simply preface it with "window." to work around jshint.

hgweb: define locally used variables as actually local in mercurial.js Variables that are used or assigned without any declaration using var (or let, or const) are considered global. In many cases this is inadvertent and actually causes a variable leaking to a broader scope, such as a temporary variable used inside a loop suddenly being accessible in global scope. (This corresponds to "undef" option of jshint). So this patch limits the scope of variables that don't need to be global. There are a lot of helper variables in Graph.render() used in a loop, I've declared them all on one line to reduce patch size. "radius" is special because it wasn't passed to graph.vertex, but was used there (it worked because this variable leaked to global scope). "window.graph" is created by an inline script in graph.tmpl so that it can be used in ajaxScrollInit() function, this patch makes this fact explicit by assigning window.graph to a local variable.

hgweb: rename an instance of XMLHttpRequest to xhr in mercurial.js "xhr" is a really widespread name for this kind of things, no idea where did "xfr" come from (the original 2228bd109706 doesn't explain that). Let's just change one letter so the name makes more sense.

hgweb: properly iterate over arrays and objects in mercurial.js In JavaScript, using for-in loops to access every property of an object can have unexpected results when inheritance is involved. For example, if some piece of code adds a property (it may be a method too) to Object.prototype, then all for-in loops that iterate over keys of any object (also anything that inherits Object) will get that property on one of the iterations. To filter out such unexpected properties for-in loops have to use Object.hasOwnProperty() method. (This corresponds to "forin" option of jshint). In the two first cases "data" and "edges" are arrays, to it's simpler to just switch to using a regular for-with-a-counter loop.

hgweb: use strict equals in mercurial.js This patch changes "==" (equals operator) to "===" (strict equals operator). The difference between them is that the latter doesn't do any type coercions. It's handy to compare string '1' to number 1 sometimes, but most of the time using "==" is inadvertent and can be replaced by an explicit type conversion. (This corresponds to "eqeqeq" option of jshint). Some of the changes in this patch are straightforward, e.g. when comparing results of typeof (they could only be strings). The same goes for 'none' and similar strings that can't be sensibly coerced to some other type. Two changes that compare values to "1" and "0" can be clarified: getAttribute() returns either a string or null, but comparing null to a string is always false, so no logic is lost.

hgweb: use strict equals, remove non-breaking space in followlines.js The first hunk had a non-breaking space character just before "{", it's not an error or anything, but let's fix it while we're at it. (This corresponds to "nonbsp" option of jshint). Hunks 2 and 3 change "==" (equals operator) to "===" (strict equals operator). The difference between them is that the latter doesn't do any type coercions. It's handy to compare string '1' to number 1 sometimes, but most of the time using "==" is inadvertent and can be replaced by an explicit type conversion. (This corresponds to "eqeqeq" option of jshint). Most of this file already uses strict equals operator, and in the code affected type coercion is not needed, because tagName and selectableTag are both strings and endId and startId are both numbers.

run-tests: make "| foo (re)" not match everything We make "foo (re)" match the entire line by adding a \Z to the regular expression before matching. However, that doesn't help when the regular expression is something like "| foo", because that gets translated to "| foo\Z", where the "|" has lower precedence and it thus matches the empty string, which is of course a prefix of every string. Fix by wrapping expression in a group before adding the \Z to the end. Differential Revision: https://phab.mercurial-scm.org/D1546

tests: fix regex in test-subrepo-git.t to match entire string Due to a bug in the test runner (fixed by the next commit), the regex used for matching lines like " foobar | 2 +-" stoppped at the "|" and the test passed even though the rest of the line did not match. The test seems to have been supposed to match "|" and "+" literally on those lines, so this changes the regex to escape those characters. It also changes a "\s*" to "\s+" since I think we'll always include a space after the "|" in the diffstat output. Differential Revision: https://phab.mercurial-scm.org/D1545

contrib: improve check-code ban on $LOCALIP in output without (glob) Differential Revision: https://phab.mercurial-scm.org/D1553

tests: re-add (glob) for $LOCALIP matches This should fix most of the failing tests on the FreeBSD builder, since it has no 127/8 series IP as a side effect of being trapped in a jail. Differential Revision: https://phab.mercurial-scm.org/D1552

py3: make sure the first argument of time.strftime() is str time.strftime() does not accepts bytes as its first argument on py3. Differential Revision: https://phab.mercurial-scm.org/D1559

py3: alias xrange to range in tests/seq.py Differential Revision: https://phab.mercurial-scm.org/D1560

py3: use pycompat.maplist() instead of map() Differential Revision: https://phab.mercurial-scm.org/D1558

tests: move JSON escape test to test-hgweb-json.t The original tests (kanji and null) in test-hgweb-commands.t come from aff419e260f9 and 823a7d79ef82, but they check json escape filter by using JavaScript variable on /graph page, which is awkward, and I'm planning to remove commit description from this variable soon. Let's move the parts that check json template filter to a more appropriate file and use normal json-* templates.

py3: fix handling of keyword arguments in revert Differential Revision: https://phab.mercurial-scm.org/D1554

fsmonitor: issue debug messages when we fall back to core status Having more information about when and why fsmonitor bails out help when looking into status performance.

py3: add b'' to regular expressions which are raw strings Differential Revision: https://phab.mercurial-scm.org/D1538

py3: use '%d' for integers rather than '%s' obsolete._readmarkers() returns an integer version number. Differential Revision: https://phab.mercurial-scm.org/D1537

py3: fix args handling for obsfate template Differential Revision: https://phab.mercurial-scm.org/D1536

py3: remove test-terse-status.t from python3 whitelist as it was renamed The renamed file exists in the whitelist. Differential Revision: https://phab.mercurial-scm.org/D1540

test-lfs: allow the test server to be killed on Windows Apparently '$!' doesn't return a Win32 PID, so the process was never killed, and the next run was screwed up. Oddly, without the explicit killdaemons.py at the end, the test seems to hang. This spawning is just sad, so I limited it to Windows.

test-lfs: perform the `chmod +x` command in a manner compatible with Windows

hghave: add a check for lfs-test-server This is consistent with how the other tests require a feature.

hgweb: show changeset age in more places (gitweb and monoblue) mercurial.js has a process_dates() function that calculates relative age for a given date, it works for all elements with "age" css class. If those elements also have "date" css class, the original text is preserved and age is added at the end. This patch adds these two css classes in some pages in gitweb and monoblue that weren't already using this feature.

obsolete: drop usage of changectx in '_computecontentdivergentset' Changectx are expensive and not needed there. The use of `repo.set` denote old code that predate the introduction of `repo.revs` that we now use. On my mercurial repository 495 draft: before: 0.054239 second after: 0.046935 second On a mercurial repository with 115973 draft: before: 0.564548 second after: 0.130534 second

obsolete: drop usage of changectx in '_computephasedivergentset' Changectx are expensive and not needed there. The use of `repo.set` denote old code that predate the introduction of `repo.revs` that we now use. On my mercurial repository 495 draft: before: 0.010275 second after: 0.008832 second On a mercurial repository with 115973 draft: before: 0.899255 second after: 0.397131 second

hgweb: remove unused Graph() properties Both of these were introduced in 0dba955c2636, but were already unused.

gitweb: remove unused css classes Looks like they were unused since the very introduction of gitweb theme in 385b8872b8e3.

monoblue: also highlight target line on annotate and comparison pages Clicking on a line link on pages that show any kind of file contents (including diffs) should highlight that line, and in monoblue it works when there's a <pre> element (e.g. diff), but pages that use <table> element (annotate and compare) need this css class. It matches and highlights linked (":target") table rows. This line is pretty much copied from gitweb theme.

paper: remove css hack that made .branchname look like .branchhead There's a visual difference in hgweb between one changeset that is the tip of its branch and another that simply belongs to that branch. But paper theme ignored this difference on changeset page and used to always use "branchname" css class, be that changeset the tip of its branch or not. That has been recently fixed, so this piece of css is not needed anymore.

hgweb: rename the main attribute of instabilities Let's make "instabilities" list contain items with "instability" key as opposed to "name" key. This way it's more explicit and more consistent with the log command-line template.

hbisect: use a defaultdict to avoid large allocations for a large changelogs We can avoid a SPACE(len(changelog)) allocation by using a defaultdict. Test Plan: python run-tests.py test-bisect* Differential Revision: https://phab.mercurial-scm.org/D1499

hbisect: use a revset for ancestor calculation Since we have revsets we can be more concise in doing the ancestor calulcation. Significant commits are all descendent of the topmost good commits. Test Plan: python run-tests.py test-bisect* Differential Revision: https://phab.mercurial-scm.org/D1498

hbisect: pass repo into hbisect.bisect Pass repo into the bisect function to get more flexibility in what we can call. This will allow us to use revsets to rewrite parts of the ancestor and children calculation in later patches. Test Plan: python run-tests.py test-bisect* Differential Revision: https://phab.mercurial-scm.org/D1497

develwarn: do not emit warning if "config" is unspecified Previously, if the develwarn call site did not specify the category of warning, and devel.all-warnings was False, it would emit the warning. If it was intended that this emit a warning if config is unspecified, I would have expected a comment, so I assumed this was unintentional and am changing the behavior. Differential Revision: https://phab.mercurial-scm.org/D1494

histedit: add support to output nodechanges using formatter The JSON output of nodechanges will help in automation and to improve editor integrations such as for Nuclide. Differential Revision: https://phab.mercurial-scm.org/D1348

rebase: use fm.formatlist() and fm.formatdict() to support user template Thanks to Yuya for suggesting this in D1173. Differential Revision: https://phab.mercurial-scm.org/D1293

tests: add test for rebase template showing wrong behavior The output for template "{nodechanges % '{key}:{value}'}" does not contain the output for value. Differential Revision: https://phab.mercurial-scm.org/D1462

remove: print message for each file in verbose mode only while using `-A` (BC) hg rm -A option prints the message of every file in the repo. This is not very user friendly for a big repository with thousands of files. So enabling this feature only when run in --verbose mode (hg rm -Av) Differential Revision: https://phab.mercurial-scm.org/D1336

localrepo: update comments around path auditors

localrepo: specify optional callback parameter to pathauditor as a keyword

bundle2: avoid unbound read when seeking Currently, seekableunbundlepart.seek() will perform a read() during seek operations. This will allocate a buffer to hold the raw data over the seek distance. This can lead to very large allocations and cause performance to suffer. We change the code to perform read(32768) in a loop to avoid potentially large allocations. `hg perfbundleread` on an uncompressed Firefox bundle reveals a performance impact: ! bundle2 iterparts() ! wall 2.992605 comb 2.990000 user 2.260000 sys 0.730000 (best of 4) ! bundle2 iterparts() seekable ! wall 3.863810 comb 3.860000 user 3.000000 sys 0.860000 (best of 3) ! bundle2 part seek() ! wall 6.213387 comb 6.200000 user 3.350000 sys 2.850000 (best of 3) ! wall 3.820347 comb 3.810000 user 2.980000 sys 0.830000 (best of 3) Since seekable bundle parts are (only) used by bundlerepo, this /may/ speed up initial loading of bundle-based repos. But any improvement will likely only be noticed on very large bundles. Differential Revision: https://phab.mercurial-scm.org/D1394

bundle2: inline struct operations Before, we were calling struct.unpack() (via an alias) on every loop iteration. I'm not sure what Python does under the hood, but it would have to look at the struct format and determine what to do. This commit establishes a struct.Struct instance and reuses it for struct reading. We can see the impact from running `hg perfbundleread` on a Firefox bundle: ! read(8k) ! wall 0.679730 comb 0.680000 user 0.140000 sys 0.540000 (best of 15) ! read(16k) ! wall 0.577228 comb 0.570000 user 0.080000 sys 0.490000 (best of 17) ! read(32k) ! wall 0.516060 comb 0.520000 user 0.040000 sys 0.480000 (best of 20) ! read(128k) ! wall 0.496378 comb 0.490000 user 0.010000 sys 0.480000 (best of 20) ! bundle2 iterparts() ! wall 3.056811 comb 3.050000 user 2.340000 sys 0.710000 (best of 4) ! wall 2.992605 comb 2.990000 user 2.260000 sys 0.730000 (best of 4) ! bundle2 iterparts() seekable ! wall 4.007676 comb 4.000000 user 3.170000 sys 0.830000 (best of 3) ! wall 3.863810 comb 3.860000 user 3.000000 sys 0.860000 (best of 3) ! bundle2 part seek() ! wall 6.267110 comb 6.250000 user 3.480000 sys 2.770000 (best of 3) ! wall 6.213387 comb 6.200000 user 3.350000 sys 2.850000 (best of 3) ! bundle2 part read(8k) ! wall 3.404164 comb 3.400000 user 2.650000 sys 0.750000 (best of 3) ! wall 3.241099 comb 3.250000 user 2.560000 sys 0.690000 (best of 3) ! bundle2 part read(16k) ! wall 3.197972 comb 3.200000 user 2.490000 sys 0.710000 (best of 4) ! wall 3.003930 comb 3.000000 user 2.270000 sys 0.730000 (best of 4) ! bundle2 part read(32k) ! wall 3.060557 comb 3.060000 user 2.340000 sys 0.720000 (best of 4) ! wall 2.904695 comb 2.900000 user 2.160000 sys 0.740000 (best of 4) ! bundle2 part read(128k) ! wall 2.952209 comb 2.950000 user 2.230000 sys 0.720000 (best of 4) ! wall 2.776140 comb 2.780000 user 2.070000 sys 0.710000 (best of 4) Profiling now says most remaining time is spent in util.chunkbuffer. I already heavily optimized that data structure several releases ago. So we'll likely get little more performance out of bundle2 reading while still retaining util.chunkbuffer(). Differential Revision: https://phab.mercurial-scm.org/D1393

bundle2: inline changegroup.readexactly() Profiling reveals this loop is pretty tight. Literally any function call elimination can make a big difference. This commit inlines the relatively trivial changegroup.readexactly() method inside the loop. The results with `hg perfbundleread` on a bundle of the Firefox repo speak for themselves: ! read(8k) ! wall 0.679730 comb 0.680000 user 0.140000 sys 0.540000 (best of 15) ! read(16k) ! wall 0.577228 comb 0.570000 user 0.080000 sys 0.490000 (best of 17) ! read(32k) ! wall 0.516060 comb 0.520000 user 0.040000 sys 0.480000 (best of 20) ! read(128k) ! wall 0.496378 comb 0.490000 user 0.010000 sys 0.480000 (best of 20) ! bundle2 iterparts() ! wall 3.460903 comb 3.460000 user 2.760000 sys 0.700000 (best of 3) ! wall 3.056811 comb 3.050000 user 2.340000 sys 0.710000 (best of 4) ! bundle2 iterparts() seekable ! wall 4.312722 comb 4.310000 user 3.480000 sys 0.830000 (best of 3) ! wall 4.007676 comb 4.000000 user 3.170000 sys 0.830000 (best of 3) ! bundle2 part seek() ! wall 6.754764 comb 6.740000 user 3.970000 sys 2.770000 (best of 3) ! wall 6.267110 comb 6.250000 user 3.480000 sys 2.770000 (best of 3) ! bundle2 part read(8k) ! wall 3.668004 comb 3.660000 user 2.960000 sys 0.700000 (best of 3) ! wall 3.404164 comb 3.400000 user 2.650000 sys 0.750000 (best of 3) ! bundle2 part read(16k) ! wall 3.489196 comb 3.480000 user 2.750000 sys 0.730000 (best of 3) ! wall 3.197972 comb 3.200000 user 2.490000 sys 0.710000 (best of 4) ! bundle2 part read(32k) ! wall 3.388569 comb 3.380000 user 2.640000 sys 0.740000 (best of 3) ! wall 3.060557 comb 3.060000 user 2.340000 sys 0.720000 (best of 4) ! bundle2 part read(128k) ! wall 3.276415 comb 3.270000 user 2.560000 sys 0.710000 (best of 4) ! wall 2.952209 comb 2.950000 user 2.230000 sys 0.720000 (best of 4) Differential Revision: https://phab.mercurial-scm.org/D1392

bundle2: inline debug logging Profiling revealed that repeated calls to indebug() were consuming a fair amount of CPU during bundle2 reading, with most of the time spent in ui.configbool(). Inlining indebug() and avoiding extra attribute lookups speeds things up substantially. Using `hg perfbundleread` with a Firefox bundle: ! read(8k) ! wall 0.679730 comb 0.680000 user 0.140000 sys 0.540000 (best of 15) ! read(16k) ! wall 0.577228 comb 0.570000 user 0.080000 sys 0.490000 (best of 17) ! read(32k) ! wall 0.516060 comb 0.520000 user 0.040000 sys 0.480000 (best of 20) ! read(128k) ! wall 0.496378 comb 0.490000 user 0.010000 sys 0.480000 (best of 20) ! bundle2 iterparts() ! wall 6.983756 comb 6.980000 user 6.220000 sys 0.760000 (best of 3) ! wall 3.460903 comb 3.460000 user 2.760000 sys 0.700000 (best of 3) ! bundle2 iterparts() seekable ! wall 8.132131 comb 8.110000 user 7.160000 sys 0.950000 (best of 3) ! wall 4.312722 comb 4.310000 user 3.480000 sys 0.830000 (best of 3) ! bundle2 part seek() ! wall 10.860942 comb 10.840000 user 7.790000 sys 3.050000 (best of 3) ! wall 6.754764 comb 6.740000 user 3.970000 sys 2.770000 (best of 3) ! bundle2 part read(8k) ! wall 7.258035 comb 7.260000 user 6.470000 sys 0.790000 (best of 3) ! wall 3.668004 comb 3.660000 user 2.960000 sys 0.700000 (best of 3) ! bundle2 part read(16k) ! wall 7.099891 comb 7.080000 user 6.310000 sys 0.770000 (best of 3) ! wall 3.489196 comb 3.480000 user 2.750000 sys 0.730000 (best of 3) ! bundle2 part read(32k) ! wall 6.964685 comb 6.950000 user 6.130000 sys 0.820000 (best of 3) ! wall 3.388569 comb 3.380000 user 2.640000 sys 0.740000 (best of 3) ! bundle2 part read(128k) ! wall 6.852867 comb 6.850000 user 6.060000 sys 0.790000 (best of 3) ! wall 3.276415 comb 3.270000 user 2.560000 sys 0.710000 (best of 4) Differential Revision: https://phab.mercurial-scm.org/D1391

bundle2: don't use seekable bundle2 parts by default (issue5691) The last commit removed the last use of the bundle2 part seek() API in the generic bundle2 part iteration code. This means we can now switch to using unseekable bundle2 parts by default and have the special consumers that actually need the behavior request it. This commit changes unbundle20.iterparts() to expose non-seekable unbundlepart instances by default. If seekable parts are needed, callers can pass "seekable=True." The bundlerepo class needs seekable parts, so it does this. The interrupt handler is also changed to use a regular unbundlepart. So, by default, all consumers except bundlerepo will see unseekable parts. Because the behavior of the iterparts() benchmark changed, we add a variation to test seekable parts vs unseekable parts. And because parts no longer have seek() unless "seekable=True," we update the "part seek" benchmark. Speaking of benchmarks, this change has the following impact to `hg perfbundleread` on an uncompressed bundle of the Firefox repo (6,070,036,163 bytes): ! read(8k) ! wall 0.722709 comb 0.720000 user 0.150000 sys 0.570000 (best of 14) ! read(16k) ! wall 0.602208 comb 0.590000 user 0.080000 sys 0.510000 (best of 17) ! read(32k) ! wall 0.554018 comb 0.560000 user 0.050000 sys 0.510000 (best of 18) ! read(128k) ! wall 0.520086 comb 0.530000 user 0.020000 sys 0.510000 (best of 20) ! bundle2 forwardchunks() ! wall 2.996329 comb 3.000000 user 2.300000 sys 0.700000 (best of 4) ! bundle2 iterparts() ! wall 8.070791 comb 8.060000 user 7.180000 sys 0.880000 (best of 3) ! wall 6.983756 comb 6.980000 user 6.220000 sys 0.760000 (best of 3) ! bundle2 iterparts() seekable ! wall 8.132131 comb 8.110000 user 7.160000 sys 0.950000 (best of 3) ! bundle2 part seek() ! wall 10.370142 comb 10.350000 user 7.430000 sys 2.920000 (best of 3) ! wall 10.860942 comb 10.840000 user 7.790000 sys 3.050000 (best of 3) ! bundle2 part read(8k) ! wall 8.599892 comb 8.580000 user 7.720000 sys 0.860000 (best of 3) ! wall 7.258035 comb 7.260000 user 6.470000 sys 0.790000 (best of 3) ! bundle2 part read(16k) ! wall 8.265361 comb 8.250000 user 7.360000 sys 0.890000 (best of 3) ! wall 7.099891 comb 7.080000 user 6.310000 sys 0.770000 (best of 3) ! bundle2 part read(32k) ! wall 8.290308 comb 8.280000 user 7.330000 sys 0.950000 (best of 3) ! wall 6.964685 comb 6.950000 user 6.130000 sys 0.820000 (best of 3) ! bundle2 part read(128k) ! wall 8.204900 comb 8.150000 user 7.210000 sys 0.940000 (best of 3) ! wall 6.852867 comb 6.850000 user 6.060000 sys 0.790000 (best of 3) The significant speedup is due to not incurring the overhead to track payload offset data. Of course, this overhead is proportional to bundle2 part size. So a multiple gigabyte changegroup part is on the extreme side of the spectrum for real-world impact. In addition to the CPU efficiency wins, not tracking offset data also means not using memory to hold that data. Using a bundle based on the example BSD repository in issue 5691, this change has a drastic impact to memory usage during `hg unbundle` (`hg clone` would behave similarly). Before, memory usage incrementally increased for the duration of bundle processing. In other words, as we advanced through the changegroup and bundle2 part, we kept allocating more memory to hold offset data. After this change, we still increase memory during changegroup application. But the rate of increase is significantly slower. (A bulk of the remaining gradual increase appears to be the storing of revlog sizes in the transaction object to facilitate rollback.) The RSS at the end of filelog application is as follows: Before: ~752 MB After: ~567 MB So, we were storing ~185 MB of offset data that we never even used. Talk about wasteful! .. api:: bundle2 parts are no longer seekable by default. .. perf:: bundle2 read I/O throughput significantly increased. .. perf:: Significant memory use reductions when reading from bundle2 bundles. On the BSD repository, peak RSS during changegroup application decreased by ~185 MB from ~752 MB to ~567 MB. Differential Revision: https://phab.mercurial-scm.org/D1390

bundle2: only seek to beginning of part in bundlerepo For reasons still not yet fully understood by me, bundlerepo requires its changegroup bundle2 part to be seeked to beginning after part iteration. As far as I can tell, it is the only bundle2 part consumer that relies on this behavior. This seeking was performed in the generic iterparts() API. Again, I don't fully understand why it was here and not in bundlerepo. Probably historical reasons. What I do know is that all other bundle2 part consumers don't need this special behavior (assuming the tests are comprehensive). So, we move the code from bundle2's iterparts() to bundlerepo's consumption of iterparts(). Differential Revision: https://phab.mercurial-scm.org/D1389

bundle2: implement consume() API on unbundlepart We want bundle parts to not be seekable by default. That means eliminating the generic seek() method. A common pattern in bundle2.py is to seek to the end of the part data. This is mainly used by the part iteration code to ensure the underlying stream is advanced to the next bundle part. In this commit, we establish a dedicated API for consuming a bundle2 part data. We switch users of seek() to it. The old implementation of seek(0, os.SEEK_END) would effectively call self.read(). The new implementation calls self.read(32768) in a loop. The old implementation would therefore assemble a buffer to hold all remaining data being seeked over. For seeking over large bundle parts, this would involve a large allocation and a lot of overhead to collect intermediate data! This overhead can be seen in the results for `hg perfbundleread`: ! bundle2 iterparts() ! wall 10.891305 comb 10.820000 user 7.990000 sys 2.830000 (best of 3) ! wall 8.070791 comb 8.060000 user 7.180000 sys 0.880000 (best of 3) ! bundle2 part seek() ! wall 12.991478 comb 10.390000 user 7.720000 sys 2.670000 (best of 3) ! wall 10.370142 comb 10.350000 user 7.430000 sys 2.920000 (best of 3) Of course, skipping over large payload data isn't likely very common. So I doubt the performance wins will be observed in the wild. Differential Revision: https://phab.mercurial-scm.org/D1388

bundle2: implement generic part payload decoder The previous commit extracted _payloadchunks() to a new derived class. There was still a reference to this method in unbundlepart, making unbundlepart unusable on its own. This commit implements a generic version of a bundle2 part payload decoder, without offset tracking. seekableunbundlepart._payloadchunks() has been refactored to consume it, adding offset tracking like before. We also implement unbundlepart._payloadchunks(), which is a thin wrapper for it. Since we never instantiate unbundlepart directly, this new method is not used. This will be changed in subsequent commits. The new implementation also inlines some simple code from unpackermixin and adds some local variable to prevent extra function calls and attribute lookups. `hg perfbundleread` on an uncompressed Firefox bundle seems to show a minor win: ! bundle2 iterparts() ! wall 12.593258 comb 12.250000 user 8.870000 sys 3.380000 (best of 3) ! wall 10.891305 comb 10.820000 user 7.990000 sys 2.830000 (best of 3) ! bundle2 part seek() ! wall 13.173163 comb 11.100000 user 8.390000 sys 2.710000 (best of 3) ! wall 12.991478 comb 10.390000 user 7.720000 sys 2.670000 (best of 3) ! bundle2 part read(8k) ! wall 9.483612 comb 9.480000 user 8.420000 sys 1.060000 (best of 3) ! wall 8.599892 comb 8.580000 user 7.720000 sys 0.860000 (best of 3) ! bundle2 part read(16k) ! wall 9.159815 comb 9.150000 user 8.220000 sys 0.930000 (best of 3) ! wall 8.265361 comb 8.250000 user 7.360000 sys 0.890000 (best of 3) ! bundle2 part read(32k) ! wall 9.141308 comb 9.130000 user 8.220000 sys 0.910000 (best of 3) ! wall 8.290308 comb 8.280000 user 7.330000 sys 0.950000 (best of 3) ! bundle2 part read(128k) ! wall 8.880587 comb 8.850000 user 7.960000 sys 0.890000 (best of 3) ! wall 8.204900 comb 8.150000 user 7.210000 sys 0.940000 (best of 3) Function call overhead in Python strikes again! Of course, bundle2 decoding CPU overhead is likely small compared to decompression and changegroup application. But every little bit helps. Differential Revision: https://phab.mercurial-scm.org/D1387

bundle2: extract logic for seeking bundle2 part into own class Currently, unbundlepart classes support bi-directional seeking. Most consumers of unbundlepart only ever seek forward - typically as part of moving to the end of the bundle part so they can move on to the next one. But regardless of the actual usage of the part, instances maintain an index mapping offsets within the underlying raw payload to offsets within the decoded payload. Maintaining the mapping of offset data can be expensive in terms of memory use. Furthermore, many bundle2 consumers don't have access to an underlying seekable stream. This includes all compressed bundles. So maintaining offset data when the underlying stream can't be seeked anyway is wasteful. And since many bundle2 streams can't be seeked, it seems like a bad idea to expose a seek API in bundle2 parts by default. If you provide them, people will attempt to use them. Seekable bundle2 parts should be the exception, not the rule. This commit starts the process dividing unbundlepart into 2 classes: a base class that supports linear, one-time reads and a child class that supports bi-directional seeking. In this first commit, we split various methods and attributes out into a new "seekableunbundlepart" class. Previous instantiators of "unbundlepart" now instantiate "seekableunbundlepart." This preserves backwards compatibility. The coupling between the classes is still tight: "unbundlepart" cannot be used on its own. This will be addressed in subsequent commits. Differential Revision: https://phab.mercurial-scm.org/D1386

merge with i18n

i18n-pt_BR: synchronized with cabc840ffdee

perf: add command to benchmark bundle reading Upcoming commits will be refactoring bundle2 I/O code. This commit establishes a `hg perfbundleread` command that measures how long it takes to read a bundle using various mechanisms. As a baseline, here's output from an uncompressed bundle1 bundle of my Firefox repo (7,098,622,890 bytes): ! read(8k) ! wall 0.763481 comb 0.760000 user 0.160000 sys 0.600000 (best of 6) ! read(16k) ! wall 0.644512 comb 0.640000 user 0.110000 sys 0.530000 (best of 16) ! read(32k) ! wall 0.581172 comb 0.590000 user 0.060000 sys 0.530000 (best of 18) ! read(128k) ! wall 0.535183 comb 0.530000 user 0.010000 sys 0.520000 (best of 19) ! cg1 deltaiter() ! wall 0.873500 comb 0.880000 user 0.840000 sys 0.040000 (best of 12) ! cg1 getchunks() ! wall 6.283797 comb 6.270000 user 5.570000 sys 0.700000 (best of 3) ! cg1 read(8k) ! wall 1.097173 comb 1.100000 user 0.400000 sys 0.700000 (best of 10) ! cg1 read(16k) ! wall 0.810750 comb 0.800000 user 0.200000 sys 0.600000 (best of 13) ! cg1 read(32k) ! wall 0.671215 comb 0.670000 user 0.110000 sys 0.560000 (best of 15) ! cg1 read(128k) ! wall 0.597857 comb 0.600000 user 0.020000 sys 0.580000 (best of 15) And from an uncompressed bundle2 bundle (6,070,036,163 bytes): ! read(8k) ! wall 0.676997 comb 0.680000 user 0.160000 sys 0.520000 (best of 15) ! read(16k) ! wall 0.592706 comb 0.590000 user 0.080000 sys 0.510000 (best of 17) ! read(32k) ! wall 0.529395 comb 0.530000 user 0.050000 sys 0.480000 (best of 16) ! read(128k) ! wall 0.491270 comb 0.490000 user 0.010000 sys 0.480000 (best of 19) ! bundle2 forwardchunks() ! wall 2.997131 comb 2.990000 user 2.270000 sys 0.720000 (best of 4) ! bundle2 iterparts() ! wall 12.247197 comb 10.670000 user 8.170000 sys 2.500000 (best of 3) ! bundle2 part seek() ! wall 11.761675 comb 10.500000 user 8.240000 sys 2.260000 (best of 3) ! bundle2 part read(8k) ! wall 9.116163 comb 9.110000 user 8.240000 sys 0.870000 (best of 3) ! bundle2 part read(16k) ! wall 8.984362 comb 8.970000 user 8.110000 sys 0.860000 (best of 3) ! bundle2 part read(32k) ! wall 8.758364 comb 8.740000 user 7.860000 sys 0.880000 (best of 3) ! bundle2 part read(128k) ! wall 8.749040 comb 8.730000 user 7.830000 sys 0.900000 (best of 3) We already see some interesting data. Notably that bundle2 has significant overhead compared to bundle1. This matters for e.g. stream clone bundles, which can be applied at >1Gbps. Differential Revision: https://phab.mercurial-scm.org/D1385

sshpeer: add a configurable hint for the ssh error message Adding a possibility to configure error hint to be shown in the case of problems with SSH. Example of such hint can be "Please see http://company/internalwiki/ssh.html". Test Plan: - Ran hg pull with broken link and verified the output has no hint by default: ``` pulling from ssh://brokenrepository.com//repo remote: ssh: Could not resolve hostname brokenrepository.com: Name or service not known abort: no suitable response from remote hg! ``` - Run hg pull --config ui.ssherrorhint="Please see http://company/internalwiki/ssh.html": ``` pulling from ssh://brokenrepository.com//repo remote: ssh: Could not resolve hostname brokenrepository.com: Name or service not known abort: no suitable response from remote hg! (Please see http://company/internalwiki/ssh.html) ``` Differential Revision: https://phab.mercurial-scm.org/D1431

docs: add args/returns docs for some cmdutil, context, and registrar functions When writing my first extension, I found it hard to figure out these functions. I figured documenting their inputs/outputs would help future authors who are new to the codebase. Differential Revision: https://phab.mercurial-scm.org/D1440

commands: add value for cmdtype argument for read only commands In the previous release we added an argument `cmdtype` to registrar.command() which is a enum and tells whether the command is read only, recoverable write or unrecoverable write command. This patch add the value of cmdtype argument for commands which are read only. Differential Revision: https://phab.mercurial-scm.org/D1468

error: add InMemoryMergeConflictsError We'll raise this exception in the merge code, and in-memory users like rebase can catch it and retry without IMM. Differential Revision: https://phab.mercurial-scm.org/D1210

lfs: generate a large file by using `python` instead of yes | head yes(1) on some systems (like gcc112) feels compelled to inform you of broken pipes, such as those triggered by head(1). This works around the problem portably.

setup: add hgext.lfs to list of Python packages This is needed for lfs to get installed. Probably could stand to go into an earlier patch, but I just want to get this stuff pushed.

test-lfs: add tests demonstrating the interaction with largefiles Obviously the original series needs to be accepted first, but there are concerns about how well these extensions will play together before proceeding. It looks like the answer is surprisingly well. There are some merge surprises (largefiles seems to combine the choice of "keep tracking as a large/normal file" with taking the content of the large/normal file) and some existing diff weirdness (largefiles diffs the standins, not the large file). Converting the repo to normal files seemlessly transitions to lfs on the fly. I didn't test going the other way, because I'm not sure why anyone would want to do that. I flagged the lack of a repo requirement after converting, because some of the unsubmitted changes I have add a requirement on commit, but this somehow misses the convert case. I flagged an issue where devel-warnings are emitted on convert, which is a separate issue.

test-lfs: cast the flags printed to an int On Windows, the flag values in the subsequent tests were printing with a 'L' suffix.

lfs: register config options I'm not sure at what point we can get rid of the deprecated options, but for the sake of making progress, they are registered too.

lfs: quiesce check-module-import warnings Specifically, 'symbol import follows non-symbol import: mercurial.i18n'

lfs: import the Facebook git-lfs client extension The purpose of this is the same as the built-in largefiles extension- to handle huge files outside of the normal storage system, generally to keep the amount of data cloned to a lower amount. There are several benefits of implementing the git-lfs protocol, instead of using the largefiles extension: - Bitbucket and Github support (and probably wider support in 3rd party hosting sites in general). [1][2] - The number of hg internals monkey patched are several orders of magnitude lower, so it will be easier to reason about and maintain. Future commands will likely just work, without requiring various wrappers. - The "standin" files are only written to the filelog, not the disk. That should avoid weird edge cases where the largefile and standin files get out of sync. [3] It also avoids the occasional printing of the "hidden" standin file in various messages. - Filesets like size() will work, even if the file isn't present. (It always says 41 bytes for largefiles, whether present or not.) The only place that I see where largefiles comes out on top is that it works with `hg serve` for simple sharing, without external infrastructure. Getting lfs-test-server working was a hassle, and took awhile to figure out. Maybe we can do something to make it work in the future. Long term, I expect that this will be highly preferred over largefiles. But if we are to recommend this to largefile users, there are some UI issues to bikeshed. Until they are resolved, I've marked this experimental, and am not putting a pointer to this in the largefiles help. The (non exhaustive) list of issues I've seen so far are: - It isn't sufficient to just enable the largefiles extension- you have to explicitly add a file with --large before it will pay attention to the configured sizes and patterns on future adds. The justification being that once you use it, you're stuck with it. I've seen people confused by this, and haven't liked it myself. But it's also saved me a few times. Should we do something like have a specific enabling config setting that must be set in the local repo config, so that enabling this extension in the user or system hgrc doesn't silently start storing lfs files? - The largefiles extension adds a repo requirement when the first largefile is committed, so that the extension must always be enabled in the future. This extension is not doing that, and since I only enabled it locally to avoid infecting other repos, I got a cryptic error about missing flag processors when I cloned. Is there no repo requirement due to shallow/narrow clone considerations (or other future advanced things)? - In the (small amount of) reading I've done about the git implementation, it seems that the files and sizes are stored in a tracked .gitattributes file. I think a tracked file for this would be extremely useful for consistency across developers, but this kind of touches on the tracked hgrc file proposal a few months back. - The git client can specify file patterns, not just sizes. - The largefiles extension has a cache directory in the local repo, but also a system wide one. We should probably implement a system wide cache too, so that multiple clones don't have to refetch the files from the server. - Jun mentioned other missing features, like SSH authentication, gc, etc. The code corresponds to c0492b73c7ef in hg-experimental. [4] The only tweaks are to load the extension in the tests with 'lfs=' instead of 'lfs=$TESTDIR/../hgext3rd/lfs', change the import in the *.py test to hgext (from hgext3rd), add the 'testedwith' declaration, and mark it experimental for now. The infinite-push, p4fastimport, and remotefilelog tests were left behind. The devel-warnings for unregistered config options are not corrected yet, nor are the import check warnings. [1] https://www.mercurial-scm.org/pipermail/mercurial/2017-November/050699.html [2] https://bitbucket.org/site/master/issues/3843/largefiles-support-bb-3903 [3] https://bz.mercurial-scm.org/show_bug.cgi?id=5738 [4] https://bitbucket.org/facebook/hg-experimental

run-tests: outputdir also has to be changed if $TESTDIR is not $PWD Following a18eef03d879, running run-tests.py from outside tests/ would lead to the creation of .testtimes and test-*.t.err in $PWD instead of $TESTDIR. This patch fixes that and updates the relevant test.

hgweb: use webutil.commonentry() for nodes (but not for jsdata yet) in /graph This makes graphdata() simpler by using existing code that gets common changeset properties for showing in hgweb. graphdata() is a nested function in graph() that prepares entries for /graph view, but there are two different lists of changesets prepared: "jsdata" for JavaScript-rendered graph and "nodes" for everything else. For "jsdata", properties "node", "user", "age" and "desc" are passed through various template filters because we don't have these filters in JavaScript, so the data has to be prepared server-side. But now that commonentry() is used for producing "nodes" list (and it doesn't apply any filters), these filters need to be added to the appropriate templates (only raw at this moment, everything else either doesn't implement graph or uses JavaScript). This is a bit of refactoring that will hopefully simplify future patches. The end result is to have /graph that only renders the actual graph with nodes and vertices in JavaScript, and the rest is done server-side. This way server-side code can focus on showing a list of changesets, which is easy because we already have /log, /shortlog, etc, and JavaScript code can be simplified, making it easier to add obsolescence graph and other features.

hgweb: check changeset's original branch in graphdata() This piece of code checks if a changeset is the tip of its branch, but as can be seen above in the context, "branch" was prepared for being displayed in hgweb by making it unicode and passing it through url.escape. It's better to use the original ctx.branch().

hgweb: show instabilities of a commit In paper, coal, gitweb and monoblue a new "tag" (or multiple, if there are many instabilities) is added to the same line that has phase, branch, etc of a changeset; in gitweb and monoblue this element has a light red background, in paper and coal the element is black and underlined. In spartan theme instabilities are shown on a separate line. While test-obsolete.t uses first(phasedivergent()) revset to pick a changeset to test, that particular changeset is also an orphan, so two different instability tags are displayed.

context: add instabilities() method to basefilectx This method is now used in webutils.commonentry(), which adds common data items (commit hash, author, date, etc) for rendering changesets in hgweb. Usually, commonentry() is given a changectx as ctx; but in views related to files (e.g. file view, diff, annotate) it's replaced by a filectx, so the latter also needs to have instabilities() method.

run-test: drop 'execfile' usage for 'common-pattern.py' file This is required for Python 3.

run-test: use byte for 'common-pattern.py' path This is required for Python 3.

templates: reword 'back to filelog' link anchor text This anchor text is problematic in two ways: first, the "back to" part assumes that you got to the page showing it via the filelog. This is not necessarily true, as there are other ways to get to that view besides the filelog view, such as for example following the history of lines from a file. Second, it uses "filelog" jargon, which refers to how each file has its own revlog. This is internal jargon that has no business being exposed to the end user. I just reworded this template to improve understanding.

hgweb: show obsolescence status of a commit As with phases, spartan theme shows a simple "obsolete: yes" on its own line (this allows replacing "yes" with something more useful in future, like output of obsfate* template functions). Everywhere else a new "tag" is added to the same line that has phase, branch, etc of a changeset; in gitweb and monoblue the element has gray background, in paper and coal the element is gray with a dashed underline.

context: add obsolete() method to basefilectx This method is now used in webutils.commonentry(), which adds common data items (commit hash, author, date, etc) for rendering changesets in hgweb. Usually, commonentry() is given a changectx as ctx; but in views related to files (e.g. file view, diff, annotate) it's replaced by a filectx, so the latter also needs to have obsolete() method.

check-code: grep's context flags don't need an extra space before number A bit of useless trivia found while researching this: OpenBSD's grep's -C has a default value (of 2) and disallows space before the argument (while -A and -B allow).

dirstate: make map implementation overridable Other implementations of dirstate will want to replace the class used for the dirstate map. Allow this to happen by making the class an attribute of the dirstate. Differential Revision: https://phab.mercurial-scm.org/D1347

fsmonitor: only access inner dirstate map if it is available As part of the dirstate refactor, fsmonitor was updated to directly access the inner map of the dirstatemap object. Dirstatemap reimplementations may not use a map like this, so only access it if it is there. Differential Revision: https://phab.mercurial-scm.org/D1346

dirstate: add explicit methods for querying directories (API) The set-like object returned by dirstate.dirs may be difficult for other implementations of the dirstate to provide, and is unnecessary as it is only ever used for __contains__. Instead, provide an explicit method for testing for a directory. .. api:: dirstate no longer provides a `dirs()` method. To test for the existence of a directory in the dirstate, use `dirstate.hasdir(dirname)`. Differential Revision: https://phab.mercurial-scm.org/D1345

dirstate: remove _droppath method This method now doesn't do very much interesting and can be removed. The updated files set can be updated where _droppath was originally called. Differential Revision: https://phab.mercurial-scm.org/D1344

dirstate: move dropping of folded filenames into the dirstate map When dropping files from the dirstate, the corresponding entry in the filefoldmap is also dropped. Move this into the dirstate map object. A future implementation of the dirstate will maintain the filefoldmap differently. Differential Revision: https://phab.mercurial-scm.org/D1343

dirstate: move management of the dirstate dirs into the dirstatemap The dirstate dirs object is owned by the map, so move management of that object there. A future implementation of the dirstate will manage the dirs object differently. Differential Revision: https://phab.mercurial-scm.org/D1342

dirstate: move management of nonnormal sets into dirstate map The dirstate map owns the nonnormal sets, and so should be the class to update them. A future implementation of the dirstate will manage these maps differently. The action of clearing ambiguous times is now entirely controlled by the dirstate map, so it moves there too. Differential Revision: https://phab.mercurial-scm.org/D1341

dirstate: add explicit methods for modifying dirstate Instead of assigning dirstatetuple objects to entries in the dirstate, move responsibility for creating tuples into the dirstatemap. Differential Revision: https://phab.mercurial-scm.org/D1340

dirstate: document dirstatemap interface Differential Revision: https://phab.mercurial-scm.org/D1380

bundlerepo: rename "bundlefilespos" variable and attribute Strictly speaking, this variable tracks offsets within the changegroup, not the bundle. While we're here, mark a class attribute as private because it is. .. api:: Rename bundlerepo.bundlerepository.bundlefilespos to _cgfilespos. Differential Revision: https://phab.mercurial-scm.org/D1384

bundlerepo: rename "bundle" arguments to "cgunpacker" "bundle" was appropriate for the bundle1 days where a bundle was a changegroup. In a bundle2 world, changegroup readers are referred to as "changegroup unpackers." Differential Revision: https://phab.mercurial-scm.org/D1383

bundlerepo: use early return I like avoiding patterns that lead to the pyramid of doom. Differential Revision: https://phab.mercurial-scm.org/D1382

test-pattern: actually update tests using the patterns We mass update the tests now. This will help the next soul touching the http protocol.

test-pattern: substitute the HTTP log timestamp too We add a pattern matching the infamous timestamp in http log. Now, we should be able to have change appears in https log without having to re-glob the whole thing over and over.

test-pattern: register the current the bundle2 capabilities string The bundle capabilites are sent with every getbundle ssh connection. Every time the protocol is updated, that string is altered. We get the part about bundle2 string replaced by $USUAL_BUNDLE2_CAPS$ so that we only have to change the substitution whenever this happens.

test-pattern: register current the bundlecaps string The bundle capabilites sent with every getbundle commands. Every time the protocol is updated, that string is altered. We get that string replace by $USUAL_BUNDLE_CAPS$ so that we only have to change the substitution whenever this happens.

test-pattern: substitute common compression list The compression list as to be matched with a glob because zstd might not be part of the option. By using a substitution for these, we won't have to re-glob them over and over.

run-tests: allow to register any arbitrary pattern for replacement We add a 'common-pattern.py' file that allow to define extra pattern. This seems a cleaner approach than editing the 'run-test.py' file over and over. In addition allowing arbitrary pattern registration will also help extension. The format used is a python file is picked out of convenience defining a list of tuple in 'substitutions' variable. This is picked out of convenience since it is dead simple to implement. The end goal is to register more pattern for Mercurial test. There are multiple common patterns that change over time. That impact is annoying. Using pattern emplacement for them would be handy. The next patches will define all the needed patterns and the last patch will mass-update the tests outputs as it was easier to do in a single pass.

crecord: fix revert -ir '.^' crash caused by 3649c3f2cd 3649c3f2cd (revert: do not reverse hunks in interactive when REV is not parent (issue5096)) changed the record "operation" for the text version but missed the curses version. Without this patch, running `hg revert -ir '.^' --config ui.interface=curses` would crash with: ProgrammingError: unexpected operation: apply Differential Revision: https://phab.mercurial-scm.org/D1381

tweakdefaults: turn on ui.statuscopies Seems obviously useful and probably off by default for historical reasons. Differential Revision: https://phab.mercurial-scm.org/D1444

run-tests: fix TESTDIR if testdescs are absolute paths Commit a18eef03d879 made TESTDIR be the location of the arguments that were passed to run-tests.py instead of just PWD. It assumed that these tests were specified using relative paths, so if pwd was /tmp/foo, and the first argument was /tmp/baz, it would set TESTDIR to /tmp/foo//tmp/baz. Differential Revision: https://phab.mercurial-scm.org/D1433

hgweb: show commit phase if it's not public In spartan theme phase is shown on its own table row, because there's no single line of "tags". Everywhere else phase is prepended to the list of "tags" of a changeset. Its element has a purple-ish color in gitweb and monoblue, and a dotted line under it and no color in paper and coal (as these themes are frugal with colors). This patch intentionally doesn't touch graph, because it needs a rewrite. I'll get to it pretty soon and in the process will add phase and everything that's still coming (e.g. obsolescence and instabilities). .. feature:: hgweb now displays phases of non-public changesets

hgweb: move changeset "tags" to a template in map file (paper and coal) This patch puts all these changeset "tags" into one template shared everywhere in paper and coal themes. But it should be noted that some of the templates had different sets of tags, in some cases it was intended, in others - most likely not. First, what's up with all these different ways to get changeset's branch. There are actually 3 ways to do it in hgweb, they can all be seen in this patch; "branches", "inbranch" and "branch". They are all lists that consist of 1 or 0 items: - "branches" has ctx.branch() if current changeset is the tip of that branch - "inbranch" has ctx.branch() if current changeset is _not_ the tip of that branch and the branch is not "default" - "branch" aka "changesetbranch" has ctx.branch() if the branch is not "default" The majority of cases (7 vs 2 + /graph) in paper theme used only option 3, which meant that "default" was never displayed. But other parts of the theme disagreed with this and used option 1 and option 2 together. For example, the default view (log) displays "default" on the branch tip (can be seen right about now on m-s.o/repo/hg), but it disappears when you click on the commit. Also, using option 3 alone meant that there was no way to tell if a changeset is the tip of its branch or not (it was always assumed that it's not, see how some css classes change from "branchname" to the correct "branchhead" in tests) -- so the two different css styles that exist in paper just for this were underused. I think this patch improves the situation, even though it changes the old (even if inconsistent) behavior. The new behavior matches that of gitweb and monoblue.

logtoprocess: clean-up old comment The comment was likely to be for runshellcommand code. Differential Revision: https://phab.mercurial-scm.org/D1425

patch: accept prefix argument to changedfiles() helper I'd like to call the function from an extension, passing both "strip" and "prefix", but it currently only accepts "strip". The only in-tree caller seems to be mq.py, which doesn't even pass "strip". Differential Revision: https://phab.mercurial-scm.org/D1413

hgweb: move changeset "tags" to a template in map file (gitweb and monoblue) Less duplication and it's also easier to add extra "tags" everywhere at once. These aren't tags as defined `hg help glossary` (hence the quotes), they are simply called that. They include branch name (in different styles if changeset is a head of that branch or not), (actual) tags and bookmarks. Good candidates to add to this list would be changeset phase and obsoletion status.

hgweb: split long lines in gitweb and monoblue (changeset summary and tags)

debugdeltachain: output information about sparse read if enabled

rebase: exclude descendants of obsoletes w/o a successor in dest (issue5300) .. feature:: Let 'hg rebase' avoid content-divergence by skipping obsolete changesets (and their descendants) when they are present in the rebase set along with one of their successors but none of their successors is in destination. In the following example, when trying to rebase 3:: onto 2, the rebase will abort with "this rebase will cause divergence from: 4": o 7 f | | o 6 e | | | o 5 d' | | x | 4 d (rewritten as 5) |/ o 3 c | | o 2 x | | o | 1 b |/ o 0 a By excluding obsolete changesets without a successor in destination (4 in the example above) and their descendants, we make rebase work in this case, thus giving: o 11 e | o 10 d' | o 9 c | o 8 b | | o 7 f | | | | x 6 e (rewritten using rebase as 11) | | | | | x 5 d' (rewritten using rebase as 10) | | | | x | 4 d | |/ | x 3 c (rewritten using rebase as 9) | | o | 2 x | | | x 1 b (rewritten using rebase as 8) |/ o 0 a where branch 4:: is left behind while branch 5:: is rebased as expected. The rationale is that users may not be interested in rebasing orphan changesets when specifying a rebase set that include them but would still want "stable" ones to be rebased. Currently, the user is suggested to allow divergence (but probably does not want it) or they must specify a rebase set excluding problematic changesets (which might be a bit cumbersome). The approach proposed here corresponds to "Option 2" in https://www.mercurial-scm.org/wiki/CEDRebase. We extend _computeobsoletenotrebased() so that it also return a set of obsolete changesets in rebase set without a successor in destination but with at least one successor in rebase set. This 'obsoletewithoutsuccessorindestination' is then stored as an attribute of rebaseruntime and used in _performrebasesubset() to: * filter out descendants of these changesets from the revisions to rebase; * issue a message about these revisions being skipped. This only occurs if 'evolution.allowdivergence' option is off and 'rebaseskipobsolete' is on.

tests: add a couple of 'hg log' calls in test-rebase-obsolete.t It helps understanding tests.

tests: conditionally display obsfate in test-rebase-obsolete log output This helps understanding and debugging.

tests: clarify a comment in test-rebase-obsolete.t

bundlerepo: rename _bundle to _cgunpacker _bundle is really a changegroup unpacker instance. Rename the variable accordingly. Differential Revision: https://phab.mercurial-scm.org/D1379

bundlerepo: assign bundle attributes in bundle type blocks It is a bit wonky to assign the same object to multiple attributes and then possibly overwrite them later. Refactor the code to use a local variable and defer attribute assignment until the final values are ready. This required passing the bundle instance to _handlebundle2part(). The only use of this method I could find is Facebook's treemanifest extension. Since it is a private method, I don't think it warrants an API callout. Differential Revision: https://phab.mercurial-scm.org/D1378

bundlerepo: make bundle and bundlefile attributes private These attributes are implementation details and shouldn't be exposed outside the class. .. api:: bundlerepo.bundlerepository.bundle and bundlerepo.bundlerepository.bundlefile are now prefixed with an underscore. Differential Revision: https://phab.mercurial-scm.org/D1377

bundlerepo: don't assume there are only two bundle classes exchange.readbundle() can return a type that represents a stream clone bundle. Explicitly handle the bundle1 type and raise a reasonable error message for unhandled bundle types. Differential Revision: https://phab.mercurial-scm.org/D1376

bundlerepo: add docstring for bundlerepository class Differential Revision: https://phab.mercurial-scm.org/D1375

bundlerepo: rename arguments to bundlerepository.__init__ To reflect what they actually are. Differential Revision: https://phab.mercurial-scm.org/D1374

bundlerepo: use suffix variable It looks like the refactor in 702a26fec3e2 attempted to establish this method argument but failed to use it. My editor caught it. Differential Revision: https://phab.mercurial-scm.org/D1373

bundlerepo: make methods agree with base class My editor was complaining about mismatches between method signatures. For methods that are implemented, we change arguments to match the base. For those that aren't, we use variable arguments because it shouldn't matter. Differential Revision: https://phab.mercurial-scm.org/D1372

bundle2: use os.SEEK_* constants Constants make code easier to read than magic numbers. I also threw in an explicit argument in a caller to further increase code comprehension. Differential Revision: https://phab.mercurial-scm.org/D1370

help: adding a topic on flags This is a short topic to explain how command-line flags can be specified. Some users have been confused by hg offerring different flag syntax than some other libraries, so it'd be nice to point them to this rather than explaining it every time. Differential Revision: https://phab.mercurial-scm.org/D1270

py3: handle keyword arguments correctly in hgext/patchbomb.py The keys of keyword arguments must be str on Python 3 which is unicode. This patch make sure we pass keyword arguments with keys are str everywhere in this file and access the keys depending on whether they are bytes or str. This patch uses pycompat.{byteskwargs|strkwargs} and somewhere it also added r'' to prevent transformer from adding a b'' over there. Differential Revision: https://phab.mercurial-scm.org/D974

tests: use jshint when available to check .js files

hgweb: fix jshint issues in mercurial.js Everything is pretty self-explanatory except the last hunk, where jshint complains: "Misleading line break before '||'; readers may interpret this as an expression boundary." There is a tweakable called "laxbreak" that allows line breaks before operators, but I think it's fine to simply join this one line and avoid extra config for now (we can cook up and add a sensible .jshintrc later).

exchange: drop unused '_getbookmarks' function The function was introduced in 8491845a75b2 in mid-November 2016 but is never used anywhere in core.

bookmark: use bundle2 debug output in one push tests Having more details about the push process will help to track changes made to the actual exchange.

pull: store binary node in pullop.remotebookmarks The internal representation of bookmark value is binary. The fact we stored 'hex' was an implementation detail from using pushkey. We move the values in 'pullop.remotebookmarks' to binary before adding a way to exchange bookmarks not based on pushkey.

config: rename allow_push to allow-push As part of ConfigConsolidationPlan [1], rename the option according to the new UI guidelines [2] and add an alias for backward compatibility. [1]: https://www.mercurial-scm.org/wiki/ConfigConsolidationPlan [2]: https://www.mercurial-scm.org/wiki/UIGuideline#adding_new_options

config: rename allowpull to allow-pull As part of ConfigConsolidationPlan [1], rename the option according to the new UI guidelines [2] and add an alias for backward compatibility. [1]: https://www.mercurial-scm.org/wiki/ConfigConsolidationPlan [2]: https://www.mercurial-scm.org/wiki/UIGuideline#adding_new_options

convert: avoid wrong lfconvert defaults by moving configitems to core The `hg lfconvert --to-normal` command uses the convert extension internally to work its magic, but that produced devel-warn messages if the convert extension wasn't loaded by the user. The test in fcd2f9b06629 (modified here) wasn't showing the warnings because the convert extension was loaded via $HGRCPATH. Most of the config options default to None/False, but 'hg.usebranchnames' and 'hg.tagsbranch' are supposed to default to True and 'default' respectively. The first iteration of this was to ui.setconfig() inside lfconvert, to force the convert extension to load. But there really is no precedent for doing this, and check-config complained that 'extensions.convert' isn't documented. Yuya suggested this alternative. This partially backs out 0d5a1175d0f9.

tr-summary: keep a weakref to the unfiltered repository Repoview can have a different life cycle, causing issue in some corner cases. The particular instance that revealed this comes from localpeer. The localpeer hold a reference to the unfiltered repository, but calling 'local()' will create an on-demand 'visible' repoview. That repoview can be garbaged collected any time. Here is a simplified step by step reproduction:: 1) tr = peer.local().transaction('foo') 2) tr.close() After (1), the repoview object is garbage collected, so weakref used in (2) point to nothing. Thanks to Sean Farley for helping raising and debugging this issue.

dispatch: verify result of early command parsing Before, early options were stripped from args, and because of this, some kind of parsing errors weren't reported. For example, $ hg ci -m -Ra file would execute "hg ci -m file" in repository "a". This patch fixes the issue by parsing early options again by real getopt-based parser, and verifying the results. If the early parsing appears wrong, hg just aborts. The current error message seems not nice, and should be improved, maybe in V2 or follow-up. Note that this isn't a security feature because we can still do anything by using shell aliases.

dispatch: convert non-list option parsed by _earlygetopt() to string So we can easily compare it with the corresponding getopt() result. There's a minor behavior change. Before, "hg --cwd ''" failed with ENOENT. But with this patch, an empty cwd is silently ignored. "hg -R ''" has always worked as such, so -R has no BC.

dispatch: add option to not strip command args parsed by _earlygetopt() This allows us to parse the original args later by full-blown getopt() in order to verify the result of the faulty early parsing. Still we need the 'strip=True' behavior for shell aliases. Note that this series is RFC because it seems to change too much to be included in stable release.

dispatch: fix early parsing of short option with value like -R=foo Before, -R=foo was parsed as '-R' 'foo', which disagrees with the standard getopt behavior.

dispatch: abort if early boolean options can't be parsed Perhaps we'll need to restrict the parsing rules of --debugger and --profile, where this patch will help us know why the --debugger option doesn't work. I have another series to extend this feature to --config/--cwd/-R, but even with that, shell aliases can be used to get around the restriction.

dispatch: stop parsing of early boolean option at "--"

dispatch: extract stub function to peek boolean command option We should at least stop parsing at "--". The 'name' argument is passed for future extension.

dispatch: do not drop unpaired argument at _earlygetopt() Before, "hg log -R" just worked.

amend: update .hgsubstate before committing a memctx (issue5677) This is a minimal copy of localrepo.commit(). As the current amend() function heavily depends on the wctx API, it wasn't easy to port it to use a separate status tuple. So for now, wctx._status is updated in-place.

subrepo: extract preprocess of repo.commit() to free function No code change other than extracting a function. Maybe we should stop mutating the status argument, but that's out of the scope of stable changes.

tests: demonstrate that .hgsubstate isn't updated on amend Loosely based on test-amend.t and test-subrepo.t

amend: do not drop missing files (issue5732) samefile() can be wrong since wctx.manifest() does not include missing files, whereas missing files should be preserved on commit.

amend: do not take untracked files as modified or clean (issue5732) fctx.exists() shouldn't be used here as it checks if a file physically exists, which may disagree with the dirstate.

tests: add more complete test for status changes on amend This demonstrates that missing/untracked files are handled incorrectly. The correct outputs are suppressed by (false !), and wrong outputs are added with (true !) instead. The setup code is copied from test-status-rev.t.

hgweb: add missing semicolons to followlines.js Minor stylistic issues caught by jshint.

gitweb: apply styles from annotate tooltip to followlines popup These new colors and styles are already used in the tooltip that gets shown when user hovers over line numbers on annotate page. The old styles, replaced in this patch, look completely unrelated to gitweb or any other hgweb theme.

paper: apply styles from annotate tooltip to followlines popup These new colors and styles are already used in the tooltip that gets shown when user hovers over line numbers on annotate page. The old styles, replaced in this patch, look completely unrelated to paper or any other hgweb theme.

pull: clarify that -u only updates linearly Differential Revision: https://phab.mercurial-scm.org/D1355

dirstate: don't remove normallookup files from nonnormalset The dirstate typically tries to keep the nonnormalset and otherparentset up-to-date when making changes to the dirstate. In the case of files marked normallookup, however, it erroneously removes the file from the nonnormalset, after _addpath had just added it. This doesn't seem to matter at the moment, as nothing relies on the nonnormalset being correct at this point, however a future re-implementations of the dirstate map will require this to be kept up-to-date correctly. Differential Revision: https://phab.mercurial-scm.org/D1339

dirstate: clear map cached properties when clearing the map dirstatemap.clear should remove all record of the files in the map. This includes removing caches of values derived from these. Differential Revision: https://phab.mercurial-scm.org/D1338

util: add util.clearcachedproperty This utility function allows clearing of the cached value set up @propertycache, if there is one. Differential Revision: https://phab.mercurial-scm.org/D1337

merge with stable