sparse-revlog: add a test checking revlog deltas for a churning file The test repository contains 5000 revisions and is therefore slow to build: five minutes with CHG, over fifteen minutes without. It is too slow to build during the test. Bundling all content produce a sizeable result, 20BM, too large to be committed. Instead, we commit a script to build the expected bundle and the test checks if the bundle is available. Any run of the script will produce the same repository content, using resulting in the same hashes. Using smaller repositories was tried, however, it misses most of the cases we are planning to improve. Having them in a 5000 repository is already nice, we usually see these case in repositories in the order of magnitude of one million revisions. This test will be very useful to check various changes strategy for building delta to store in a sparse-revlog. In this series we will focus our attention on the following metrics: The ones that will impact the final storage performance (size, space): * size of the revlog data file (".hg/store/data/*.d") * chain length info The ones that describe the deltas patterns: * number of snapshot revision (and their level) * size taken by snapshot revision (and their level)

tests: add a `tests/artifacts/` directory That directory is meant to cache large items used by tests that are slow to generate. See 'PURPOSE' file for details and next changesets for a first user.

verify: make output less confusing (issue5924) output before: "500 files, 2035 changesets, 2622 total revisions" output after: "checked 2035 changesets with 2622 changes to 500 files" new one was suggested in the comments inside the issue. Differential Revision: https://phab.mercurial-scm.org/D4476

revlog: clarify the comment attached to delta reuse The previous version was a bit complicated and referred to a deprecated configuration option.

revlog: drop duplicated code This code probably got duplicated by a rebase/evolve conflict. We drop the extra copy, the other copy is right below. This had no real effects since other logic ensure that we never test the same revision twice.

wireprotov2peer: properly format errors formatrichmessage() expects an iterable containing dicts with well-defined keys. We were passing in something else. This caused an exception. Change the code to call formatrichmessage() with the proper argument. And add a TODO to potentially emit the proper data structure from the server in the first place. Differential Revision: https://phab.mercurial-scm.org/D4441

wireprotov2peer: report exceptions in frame handling against request future Otherwise the future may never resolve, which could cause deadlock. Differential Revision: https://phab.mercurial-scm.org/D4440

httppeer: use util.readexactly() to abort on incomplete responses Plain resp.read(n) may not return exactly n bytes when we need, and to detect such cases before trying to interpret whatever has been read, we can use util.readexactly(), which raises an Abort when stream ends unexpectedly. In the first case here, readexactly() prevents a traceback with struct.error, in the second it avoids looking for invalid compression engines. In this test case, _wraphttpresponse doesn't catch the problem (presumably because it doesn't know transfer encoding), and the code continues reading the response until it gets to compression engine data. Maybe there should be checks before the execution gets there, but I'm not sure where (httplib?)

httppeer: calculate total expected bytes correctly User-facing error messages that handled httplib.IncompleteRead errors in Mercurial used to look like this: abort: HTTP request error (incomplete response; expected 3 bytes got 1) But the errors that are being handled underneath the UI look like this: IncompleteRead(1 bytes read, 3 more expected) I.e. the error actually counts total number of expected bytes minus bytes already received. Before, users could see weird messages like "expected 10 bytes got 10", while in reality httplib expected 10 _more_ bytes (20 in total).

lazyancestors: reuse __iter__ implementation in __contains__ There was a comment in the code that said "Trying to do both __iter__ and __contains__ using the same visit heap and seen set is complex enough that it slows down both. Keep them separate.". However, it seems easy and efficient to make __contains__ keep an iterator across calls. I couldn't measure any slowdown from `hg bundle --all` (which seem to call lazyancestors.__contains__ frequently). Differential Revision: https://phab.mercurial-scm.org/D4508