revlog-sparse-read: add a lower-threshold for read block size The option experimental.sparse-read.min-block-size specifies the minimal size of a deltachain span, under which it won't be split by _slicechunk.

revlog: introduce an experimental flag to slice chunks reads when too sparse Delta chains can become quite sparse if there is a lot of unrelated data between relevant pieces. Right now, revlog always reads all the necessary data for the delta chain in one single read. This can lead to a lot of unrelated data to be read (see issue5482 for more details). One can use the `experimental.maxdeltachainspan` option with a large value (or -1) to easily produce a very sparse delta chain. This change introduces the ability to slice the chunks retrieval into multiple reads, skipping large sections of unrelated data. Preliminary testing shows interesting results. For example the peak memory consumption to read a manifest on a large repository is reduced from 600MB to 250MB (200MB without maxdeltachainspan). However, the slicing itself and the multiple reads can have an negative impact on performance. This is why the new feature is hidden behind an experimental flag. Future changesets will add various parameters to control the slicing heuristics. We hope to experiment a wide variety of repositories during 4.4 and hopefully turn the feature on by default in 4.5. As a first try, the algorithm itself is prone to deep changes. However, we wish to define APIs and have a baseline to work on.

revlog: ignore empty trailing chunks when reading segments When a merge commit creates an empty diff in the revlog, its offset may still be quite far from the end of the previous chunk. Skipping these empty chunks may reduce read size significantly. In most cases, there is no gain, and in some cases, little gain. On my clone of pypy, `hg manifest` reads 65% less bytes (96140 i/o 275943) for revision 4260 by ignoring the only empty trailing diff. For revision 2229, 35% (34557 i/o 53435) Sadly, this is difficult to reproduce, as hg clone can make its own different structure every time.

phase: isolate logic to update remote phrase through bundle2 pushkey Move the logic to build bundle2 pushkey part into its dedicated function. It will help to keep the logic clear when adding support for sending phases change using 'phase-heads' part.

phase: generate a push-race detection part on push We are about to switch phase pushing from using pushkey to using a the new dedicated binary part. We introduce the push race detection on its own to help detect potential impact.

phase: introduce a new 'check:phases' part This part checks if revisions are still in the same phase as when the bundle was generated. This is similar to what 'check:heads' or 'check:updated-heads' bundle2 part achieves for changesets. We needs seems before we can move away from pushkey usage from phase since pushkey has it own built-in push-race detection.

phase: gather remote phase information in a summary object We keep useful phase information around. The data will be reused with detecting push-race in later changesets.

phase: simplify the check for issue3781 shortcut in discovery We'll rework the code around this check. Limiting the entanglement will help with later changesets

exchange: fix issue3781 reference in the comment This comment is about: issue3781: Courtesy Phases synchronisation to publishing server prevent subrepo push Not about: issue3871: Slow hg log when template contains {file_adds}, {file_mods} and {file_dels}

phase: filter out non-draft item in "draft root" The on-disk file can contain draft root that are descendants of secret root. The resulting phase computation is correct, but the phases root content is not. I will send another series to introduce code that remove some of the cases where this can happens, but we first need to damage control the existing case. After this changeset, we can no longer advertise secret changeset as draft root.

subrepo: share instead of clone if the parent repo is shared (issue5675) (BC) Previously, only the top level repo was shared, and then any subrepos were cloned on demand. This is problematic because commits to the parent repo would write an updated .hgsubstate to the share source, but the corresponding subrepo commit would be stuck in the local subrepo. That would prevent an update in the source repo. We already go to great lengths to avoid having inconsistent repos (e.g., `hg push -r rev` will push _everything_ in a subrepo, even if it isn't referenced in one of the parent's outgoing commits). Therefore, this seems like a bug fix, and there's no option to get the old behavior. I can't imagine the previous behavior was useful to anybody. There shouldn't be an issue with svn, since it is centralized. Maybe --git-dir can be used for git subrepos, but I'll leave that to someone more familiar with git. An integer was previously being implicitly returned from commands.share(), which caused dispatch() to start crashing when changing over to returning the shared repo. All error paths appear to raise, so this can be hardcoded to success. The clone command checks for 'is None' in a similar pattern, but since hg.clone() always returns a tuple, that seems wrong? .. fix:: Issue 5675 Creating a share of a repository with a Mercurial subrepository will now share the subrepository. and .. bc:: Mercurial subrepositories are now shared instead of cloned when the parent repository is shared. This prevents dangling subrepository references in the share source. Previously shared repositories with cloned subrepositories will continue to function unchanged.

tests: update output for no-symlink platforms This goes with eb586ed5d8ce.

releasenotes: show a warning if fuzzywuzzy is not present Differential Revision: https://phab.mercurial-scm.org/D1121

releasenotes: move import of fuzzywuzzy to import level This will help us in determining easily that whether fuzzywuzzy is loaded or not loaded in any of the function. Differential Revision: https://phab.mercurial-scm.org/D1120

releasenotes: don't abort is there is a bad formatted entry for releasenotes While producing releasenotes for (4.3::), releasenotes aborts with error because of some bad formatting of releasenotes in some commits. Instead of aborting, this adds warning message which will help us in skipping them and telling user about it. Differential Revision: https://phab.mercurial-scm.org/D1097

releasenotes: make the import of fuzzywuzzy optional If fuzzywuzzy is note present, we will not be having the capability to merge existing releasenotes with the new releasenotes on the similarity basis. The merging will still work good for exact same releasenotes entries. Differential Revision: https://phab.mercurial-scm.org/D1096

webcommands: replace str(ctx) etc with pycompat.bytestr(ctx) etc hgweb can now serve the graph view in Python 3. Differential Revision: https://phab.mercurial-scm.org/D1138

templater: don't blow up when trying to build an abort message __name__ is unicode, but we need bytes. For now, we'll make the (mostly-safe) assumption that template filter names will be ascii. Differential Revision: https://phab.mercurial-scm.org/D1137

webutil: use pycompat.bytestr() instead of str() Stops us from choking the templater on Python 3. With this patch applied, much of hgweb works correctly in Python 3. The notable exception is the graph page, which chokes because it gets node IDs as str instead of bytes. Differential Revision: https://phab.mercurial-scm.org/D1135

hgweb: correct an earlier error of mine - `start` should be bytes Gets hgweb very close to working with Python 3. Differential Revision: https://phab.mercurial-scm.org/D1134

config: simplify aliasing commands.update.check experimental.updatecheck was renamed into commands.update.check, use the config system to provides the fallback on the old config name instead of adding more code. Differential Revision: https://phab.mercurial-scm.org/D1117

run-tests: set HGPLAIN=1 when bisecting Otherwise a customized template may break the regex matching the bisect output. Differential Revision: https://phab.mercurial-scm.org/D949

run-tests: extract Popen logic to a single method This removes 3 lines in total LOC and makes the upcoming changes easier. Differential Revision: https://phab.mercurial-scm.org/D948

run-tests: move bisect logic to a separate method This removes 8 space indentation and makes upcoming changes easier. Differential Revision: https://phab.mercurial-scm.org/D947

mpatch: switch alignment of wrapped line from tab to spaces with clang-format I may be a weird person for liking this style, but our C style is historically nominally the Linux Kernel style, and when you configure clang-format to be kernel-ish, this is what you get. If we want to change it, we can do so by tweaking the formatter rules in the future. Differential Revision: https://phab.mercurial-scm.org/D1132

mpatch: reformat function prototypes with clang-format Differential Revision: https://phab.mercurial-scm.org/D1131

contrib: remove check-code rule about indentation We're moving towards a clang-format world, and clang-format is able to wrap argument lists with spaces reliably, while still enforcing tabs globally. Let's let clang-format do its job, and not do as much C-style enforcement with regular expressions. Differential Revision: https://phab.mercurial-scm.org/D1130

clang-format: configuration for the clang-format source formatter Differential Revision: https://phab.mercurial-scm.org/D1129

merge: allow user to halt merge on merge-tool failures Depends on D932. Call the new _onfilemergefailure function when a merge tool reports failure via a return code. Differential Revision: https://phab.mercurial-scm.org/D951

filemerge: introduce functions to halt merge flow Depends on D931. This patch introduces functions and a config option that will allow a user to halt the merge if there are failures during a file merge. These functions will be used in the next patch. Differential Revision: https://phab.mercurial-scm.org/D932

amend: add a flag `-n/--note` to store note with amend This patch utilises the functionality added in previous patches and adds a flag to amend command in hgext/amend to add a note to the amend. Since the note is stored in the obsmarker metadata, this will only be useful when obsmarker creation is enabled, otherwise this is no-op. Not adding releasenotes part as we yet don't have a functionality in core to show the note. Differential Revision: https://phab.mercurial-scm.org/D1095

cmdutil: pass metadata from amend() to cleanupnodes `commit --amend` and amend command in core and extensions rely on cmdutil.amend() for amending a commit. So the logic to add a note to amend must reside here. This patch assumes that note will be passed in opts dictionary to the function and it will be passed to cleanupnodes and then createmarkers to store the note in the obsmarker metadata. After this patch, note can be stored on an amend changeset by passing notes as a part of opts to cmdutil.amend(). Differential Revision: https://phab.mercurial-scm.org/D1094

scmutil: add capability to cleanupnodes to take obsmarker metadata This patch adds a metadata argument to cleanupnodes() which will be dict and can be passed to obsmarker.createmarkers() and can be stored on the obsmarker. In cases when obsolescence is not enabled, the metadata argument is useless. This is a step towards storing a note in amend. Differential Revision: https://phab.mercurial-scm.org/D1093

test-hgweb-annotate-whitespace: make test compatible with chg D946 fixed a bunch of tests which had the same root cause. Please see that for details. This seems to be one of the newer tests which fails because of the same reason. Test Plan: Ran the test 'test-hgweb-annotate-whitespace.t' with and without the '--chg' option. Differential Revision: https://phab.mercurial-scm.org/D1124

test-pager: make the test compatible with chg D911 tried to make this test compatible with chg but instead resulted in the test being flaky for chg. For now, disabling this test for chg because it seems difficult to fix the test. This will allow for the continuous build setup for chg. Test Plan: Ran the test 'test-pager.t' with and without the '--chg' option. Differential Revision: https://phab.mercurial-scm.org/D1128

test-obsolete-bounds-checking: make the test compatible with chg This test fails when run with chg because the error message starts with "ProgrammingError" instead of "mercurial.error.ProgrammingError". Therefore, globing the "mercurial.error." to ensure that the test is compatible with chg. Test Plan: Ran the test 'test-obsolete-bounds-checking.t' with and without the '--chg' option. Differential Revision: https://phab.mercurial-scm.org/D1127

test-dispatch: make the test compatible with chg The test is broken when run with chg because it prints a different error message when chg is running. This commit fixes the test by special casing for chg. Test Plan: Ran the test 'test-dispatch.t' with and without '--chg' option. Differential Revision: https://phab.mercurial-scm.org/D1126

histedit: remove the nonexistant config 'histeditng' D942 removed the experimental config 'histeditng'. This is a leftover which should have been removed in that commit. Therefore, this commit completes the cleanup. Test Plan: Ran all the tests. Differential Revision: https://phab.mercurial-scm.org/D1123

context: add workingfilectx.markcopied With in-memory merge, copy information needs to be stored in-memory, not in the dirstate. To make this transition easy, move the existing dirstate-based approach to workingfilectx; that way, other implementations can choose to store it somewhere else. Differential Revision: https://phab.mercurial-scm.org/D1106

merge: don't use workers in in-memory mode The worker processes can't share memory, so workers should not be used. Differential Revision: https://phab.mercurial-scm.org/D1105

filemerge: add a missing flushall() Differential Revision: https://phab.mercurial-scm.org/D1060

filemerge: store backups in the overlayworkingctx if using imm Differential Revision: https://phab.mercurial-scm.org/D1059

context: add overlayfilectx.cmp() Differential Revision: https://phab.mercurial-scm.org/D1058

filemerge: use arbitraryfilectx for backups With in-memory merge, backup files might be overlayworkingfilectxs stored in memory. But they could also be real files if the user's backup directory is outside the working dir. Rather than have two code paths everywhere, let's use arbitraryfilectx so they can be consistent. Differential Revision: https://phab.mercurial-scm.org/D1057

registrar: add support for storing the type of command in func object This patch adds support for storing the type of command which is going to run in the func object. For this it does the following: 1) Add three possible values as attributes to the registrar.command class 2) Add a new argument to registrar.command._doregister function 3) Add a new attribute cmdtype to the func object The type of command will be helpful in deciding what level of access on hidden commits it can has. Differential Revision: https://phab.mercurial-scm.org/D736

releasenotes: fix documentation of similaritycheck() The function document says that it returns true when the fragment can be merged, but if you see the function just above it which is similar(), it writes already exists thing if return value from similaritycheck() is False which is just opposite of the doc. This patch fixes that. Differential Revision: https://phab.mercurial-scm.org/D1119

configitems: register acl config section

configitems: register the 'extdata.*.diffargs' config

configitems: register the 'exdiff.opts.*' config

configitems: register the 'hostsecurity.*:verifycertsfile' config

configitems: register the 'hostsecurity.*:fingerprints' config

configitems: register the 'hostsecurity.*:ciphers' config

configitems: register the 'hostsecurity.*:minimumprotocol' config

configitems: register the test 'fakepatchtime.fakenow' config

configitems: register the test 'fakedirstatewritetime.fakenow' config

configitems: register the 'gpg' arbitraty key section

configitems: register the 'extdata' section

configitems: fix registration of extensions config The new 'itemregister' class should be used when loading the extension config. This was and oversight when itemregister was introduced.

configitems: register the test config in 'test-push-race.t'

configitems: register the test 'ui.ioerrors' config

configitems: register the test 'failafterfinalize.fail' config

configitems: register the test 'logtoprocess.foo' config

configitems: register the test 'failpush.reason' config

configitems: register the 'patchbomb.reply-to' config

configitems: register the 'patchbomb.cc' config

configitems: register the 'patchbomb.bcc' config

configitems: register the 'web.staticurl' config

configitems: register the 'experimental.uncommitondirtywdir' config

configitems: register the 'largefiles.usercache' config

configitems: register the 'largefiles.patterns' config

configitems: register the 'largefiles.minsize' config

configitem: register the 'notify.changegroup' config

configitem: register the 'notify.outgoing' config

configitem: register the 'notify.incoming' config

configitems: register the 'hgweb-paths' section

contrib-perf: register the 'parentscount' config item

contrib-perf: update the config registration There are various issue with the previous code, all hidden by the try-except clause. These are fixed now.

configitems: register the 'hostfingerprints' section

configitems: register the 'hostsecurity.minimumprotocol' config

configitems: register the 'progress.format' config

configitems: register the 'blackbox.maxfiles' config

hgweb: fix decodevaluefromheaders to always return a bytes value That's more in line with what we want, and we know it's ASCII data since that's all HTTP technically allows in headers anyway. Differential Revision: https://phab.mercurial-scm.org/D1112

hgweb: more "headers are native strs" cleanup I'll fix the decodevaluefromheaders function in an upcoming change. Differential Revision: https://phab.mercurial-scm.org/D1111

hgweb: when unpacking args from request form, convert to bytes We assume http-originated values are ASCII, which should be safe based on the RFC. Differential Revision: https://phab.mercurial-scm.org/D1110

wireproto: more strkwargs cleanup Differential Revision: https://phab.mercurial-scm.org/D1109

hgweb: more "http headers are native strs" cleanup Differential Revision: https://phab.mercurial-scm.org/D1108

wireproto: bounce kwargs to/from bytes/str as needed Differential Revision: https://phab.mercurial-scm.org/D1107

configitems: fixup default value of annotate config option It turned out that configbool is used for most of them so the registered default value was wrong.

transaction-summary: use a revset to filter obsoletes in reportnewcs() Thanks to Yuya Nishihara for suggesting.

releasenotes: add pager support when printing to the ui Differential Revision: https://phab.mercurial-scm.org/D1098

configitems: fixup default value of diff config option It turned out that configbool is used for most of them so the registered default value was wrong.

devel-warn: add 'warn-' to 'devel.empty-changegroup' config Rename 'devel.empty-changegroup' to 'devel.warn-empty- changegroup' in order to clarify it controls a warning message. No alias is installed since the previous configuration option was not documented.

peer: ensure command names are always ascii bytestrs Differential Revision: https://phab.mercurial-scm.org/D1104

httppeer: always produce native str header keys and values Differential Revision: https://phab.mercurial-scm.org/D1103

wireproto: use %d to encode int, not %s Differential Revision: https://phab.mercurial-scm.org/D1102

wireproto: use a proper exception instead of `assert False` Differential Revision: https://phab.mercurial-scm.org/D1101

wireproto: use listcomp instead of map() The latter returns a generator object on Python 3, which breaks various parts of hg that expected a list. Differential Revision: https://phab.mercurial-scm.org/D1100

dagutil: use a listcomp instead of a map() In Python 3, the map() returns a generator object instead of a list, and some parts of hg depend on this being consumable more than once or sortable in place. Differential Revision: https://phab.mercurial-scm.org/D1099

peer: when collecting method names for batch calls, bytes-ify __name__ This will explode violently if we have a non-ascii command name. That shouldn't ever happen in core, and seems unlikely even in third-party code. Regardless, it'll explode violently, so we can revisit things in the future if we need to change the encoding here. Differential Revision: https://phab.mercurial-scm.org/D1092

python3: replace im_{self,func} with __{self,func}__ globally These new names are portable back to Python 2.6. Differential Revision: https://phab.mercurial-scm.org/D1091

httppeer: extract content-type from headers using native str Differential Revision: https://phab.mercurial-scm.org/D1090

httppeer: convert request url back to bytes before inspecting it Differential Revision: https://phab.mercurial-scm.org/D1089

templater: explode if we try to emit a str Without this if branch, we infinitely recurse in _flatten, which is very confusing. Something in an hgweb template is trying to write out a string instead of a bytes on Python 3, and this at least makes it crash politely. Differential Revision: https://phab.mercurial-scm.org/D1088

hgweb: fill in content-type and content-length as native strings This lets me actually get a capabilities response from hgweb over http. Differential Revision: https://phab.mercurial-scm.org/D1087

hgweb: mimetype guessing needs a unicode path Differential Revision: https://phab.mercurial-scm.org/D1086

hgweb: set sent_headers attr as early as practical While doing Python 3 porting work, I've seen exceptions happen in parts of hgweb we normally assume are robust. It won't hurt anything to set this attribute significantly earlier, so let's do so and save confusing during the porting process. Differential Revision: https://phab.mercurial-scm.org/D1085

hgweb: detect Python 3-era libraries and use modern attribute names Differential Revision: https://phab.mercurial-scm.org/D1084

server: indent block that's about to get conditionalized Differential Revision: https://phab.mercurial-scm.org/D1083

templates: fix missed space between instability labels

templates: introduce labelcset() function in map-cmdline.default as example

templater: load aliases from [templatealias] section in map file This seems sometimes useful as an alias can be a function, but a template fragment can't.

templater: load template fragments from [templates] section in map file This allows us to %include map-cmdline.<style> file in our .hgrc files. The syntax is slightly different as hgrc doesn't support loading an external template file, but map-cmdline files don't use this feature, so the syntax can be considered identical in practice. Unnamed section is remapped for backward compatibility.

config: allow remapping the default section The next patch depends on it. It doesn't make sense that the default section can't be remapped with {'': whatever}.

templater: simplify merge of __base__ dicts by reading it first

phase: add a dedicated pretxnclose-phase hook This new hook mirror the newly introduced 'txnclose-phase' but can abort the transaction.

phase: add a dedicated txnclose-phase hook The new 'txnclose-phase' hook expose the phase movement information stored in 'tr.changes['phases]'. To provide a simple and straightforward hook API to the users, we introduce a new hook called for each revision affected. Since a transaction can affect the phase of multiple changesets, updating the existing 'txnclose' hook to expose that information would be more complex. The data for all moves will not fit in environment variables and iterations over each move would be cumbersome. So the introduction of a new dedicated hook is preferred in this changesets. This does not exclude the addition of the full phase movement information to the existing 'txnclose' in the future to help write more complex hooks.

bookmark: add a dedicated pretxnclose-bookmark hook This new hook mirror the newly introduced 'txnclose-bookmark' but can abort the transaction.

bookmark: add a dedicated txnclose-bookmark hook The new 'txnclose-bookmark' hook expose the bookmark movement information stored in 'tr.changes['bookmarks]'. To provide a simple and straightforward hook API to the users, we introduce a new hook called for each bookmark touched. Since a transaction can affect multiple bookmarks, updating the existing 'txnclose' hook to expose that information would be more complex. The data for all moves might not fit in environment variables and iterations over each move would be cumbersome. So the introduction of a new dedicated hook is preferred in this changeset. This does not exclude the addition to the full bookmark information to the existing 'txnclose' in the future to help write more complex hooks.

tweakdefaults: make commands.update.check be `noconflict` This is the second-safest option we have to offer in `updatecheck`, with `abort` being the safest and `linear` being the default. At the sprint we discussed how much `none` and `linear` make us all uncomfortable, and how we'd like to move the default behavior if we can. I'm not sure we can get away with actually changing the out of the box default behavior, but we can at *least* do this. Differential Revision: https://phab.mercurial-scm.org/D1062

hgweb: fix logging to use native strings as appropriate Kind of a tangled mess, but now logging works in both Python 2 and 3. # no-check-commit because of the interface required by Python's HTTP server code. Differential Revision: https://phab.mercurial-scm.org/D1080