wireproto: perform chunking and compression at protocol layer (API) Currently, the "streamres" response type is populated with a generator of chunks with compression possibly already applied. This puts the onus on commands to perform chunking and compression. Architecturally, I think this is the wrong place to perform this work. I think commands should say "here is the data" and the protocol layer should take care of encoding the final bytes to put on the wire. Additionally, upcoming commits will improve wire protocol support for compression. Having a central place for performing compression in the protocol transport layer will be easier than having to deal with compression at the commands layer. This commit refactors the "streamres" response type to accept either a generator or an object with "read." Additionally, the type now accepts a flag indicating whether the response is a "version 1 compressible" response. This basically identifies all commands currently performing compression. I could have used a special type for this, but a flag works just as well. The argument name foreshadows the introduction of wire protocol changes, hence the "v1." The code for chunking and compressing has been moved to the output generation function for each protocol transport. Some code has been inlined, resulting in the deletion of now unused methods.

httppeer: use compression engine API for decompressing responses In preparation for supporting multiple compression formats on the wire protocol, we need all users of the wire protocol to use compression engine APIs. This commit ports the HTTP wire protocol client to use the compression engine API. The code for handling the HTTPException is a bit hacky. Essentially, HTTPException could be thrown by any read() from the socket. However, as part of porting the API, we no longer have a generator wrapping the socket and we don't have a single place where we can trap the exception. We solve this by introducing a proxy class that intercepts read() and converts the exception appropriately. In the future, we could introduce a new compression engine API that supports emitting a generator of decompressed chunks. This would eliminate the need for the proxy class. As I said when I introduced the decompressorreader() API, I'm not fond of it and would support transitioning to something better. This can be done as a follow-up, preferably once all the code is using the compression engine API and we have a better idea of the API needs of all the consumers.

httppeer: do decompression inside _callstream The current HTTP transport protocol only compresses certain command responses and requires calls to that command to call "_callcompressable," which zlib decompresses the response transparently. Upcoming changes will enable *any* response to be compressed with varying compression formats. In order to handle this better, this commit moves the decompression bits to the main function performing the HTTP request. We introduce an underscore-prefixed argument to denote this behavior so it doesn't conflict with a named argument to a command.

keepalive: reorder header precedence There are 3 sources of headers used by this function: * The default headers defined by the URL opener * Headers that are copied on redirects * Headers that aren't copied on redirects Previously, we applied the default headers from the URL opener last. This feels wrong to me as those headers are the most low level and something built on top of the URL opener may wish to override them. So, this commit changes the order to apply them with the least precedence. While I was here, I removed a Python version test that is no longer necessary.

debuginstall: print compression engine support Since compression engines may be provided by extensions and since not all registered compression engines may be available to use, it seems useful to provide a mechanism to see the state of known compression engines. This commit teaches `hg debuginstall` to print info on known and available compression engines.

bdiff: don't check border condition in loop This is pretty much a copy of d500ddae7494, just to a different loop. The condition `p == plast` (`plast == a + len - 1`) was only true on the final iteration of the loop. So it was wasteful to check for it on every iteration. We decrease the iteration count by 1 and add an explicit check for `p == plast` after the loop. Again, we see modest wins. From the mozilla-unified repository: $ perfbdiff -m 3041e4d59df2 ! wall 0.035502 comb 0.040000 user 0.040000 sys 0.000000 (best of 100) ! wall 0.030480 comb 0.030000 user 0.030000 sys 0.000000 (best of 100) $ perfbdiff 0e9928989e9c --alldata --count 100 ! wall 4.097394 comb 4.100000 user 4.100000 sys 0.000000 (best of 3) ! wall 3.597798 comb 3.600000 user 3.600000 sys 0.000000 (best of 3) The 2nd example throws a total of ~3.3GB of data at bdiff. This change increases the throughput from ~811 MB/s to ~924 MB/s.

conflicts: make spacing consistent in conflict markers The way default marker template was defined before this patch, the spacing before dash in conflict markes was dependent on whether changeset is a tip one or not. This is a relevant part of template: '{ifeq(tags, "tip", "", "{tags} "}' If revision is a tip revision with no other tags, this would resolve to an empty string, but for revisions which are not tip and don't have any other tags, this would resolve to a single space string. In the end this causes weirdnesses like the ones you can see in the affected tests. This is a not a big deal, but double spacing may be visually less pleasant. Please note that test changes where commit hashes change are the result of marking files as resolved without removing markers.

rebase: move bookmark update to before rebase clearing Bookmark fixing should probably happen before the rebase starts to clean up, so let's move it before clearrebased. This will also help a future patch where we want to add more clear logic to the existing clear section.

setup: include a dummy $PATH in the custom environment used by build.py This is required for building with pypiwin32, the pip-installable replacement for pywin32.

shelve: move unshelve-finishing logic to a separate function Finishing unshelve involves two steps now: - stripping a changelog - aborting a transaction Obs-based shelve will not require these things, so isolating this logic into a separate function where the normal/obs-shelve branching is going to be implemented seems to be like a nice idea. Behavior-wise this change moves 'unshelvecleanup' from being between changelog stripping and transaction abortion to being after them. I don't think this has any negative effects.

shelve: move file-forgetting logic to a separate function This is just a readability improvement.

shelve: move rebasing logic to a separate function Rebasing restored shelved commit onto the right destination is done differently in traditional and obs-based unshelve: - for traditional, we just rebase it - for obs-based, we need to check whether a successor of the restored commit already exists in the destination (this might happen when unshelving twice on the same destination) This is the reason why this piece of logic should be in its own function: to not have excessive complexity in the main function.

shelve: move commit restoration logic to a separate function

shelve: move temporary commit creation to a separate function Committing working copy changes before rebasing a shelved commit on top of them is an independent piece of behavior, which fits into its own function. Similar to the previous series, this and a couple of following patches are for unshelve refactoring.

commands: print chunk type in debugrevlog Each data entry ("chunk") in a revlog has a type based on the first byte of the data. This type indicates how to interpret the data. This seems like a useful thing to be able to query through a debug command. So let's add that to `hg debugrevlog`. This does make `hg debugrevlog` slightly slower, as it has to read more than just the index. However, even on the mozilla-unified manifest (which is ~200MB spread over ~350K revisions), this takes <400ms.