wireprotov2: define and implement "manifestdata" command The added command can be used for obtaining manifest data. Given a manifest path and set of manifest nodes, data about manifests can be retrieved. Unlike changeset data, we wish to emit deltas to describe manifest revisions. So the command uses the relatively new API for building delta requests and emitting them. The code calls into deltaparent(), which I'm not very keen of. There's still work to be done in delta generation land so implementation details of storage (e.g. exactly one delta is stored/available) don't creep into higher levels. But we can worry about this later (there is already a TODO on imanifestorage tracking this). On the subject of parent deltas, the server assumes parent revisions exist on the receiving end. This is obviously wrong for shallow clone. I've added TODOs to add a mechanism to the command to allow clients to specify desired behavior. This shouldn't be too difficult to implement. Another big change is that the client must explicitly request manifest nodes to retrieve. This is a major departure from "getbundle," where the server derives relevant manifests as it iterates changesets and sends them automatically. As implemented, the client must transmit each requested node to the server. At 20 bytes per node, we're looking at 2 MB per 100,000 nodes. Plus wire encoding overhead. This isn't ideal for clients with limited upload bandwidth. I plan to address this in the future by allowing alternate mechanisms for defining the revisions to retrieve. One idea is to define a range of changeset revisions whose manifest revisions to retrieve (similar to how "changesetdata" works). We almost certainly want an API to look up an individual manifest by node. And that's where I've chosen to start with the implementation. Again, a theme of this early exchangev2 work is I want to start by building primitives for accessing raw repository data first and see how far we can get with those before we need more complexity. Differential Revision: https://phab.mercurial-scm.org/D4488

wireprotov2: add TODOs around extending changesetdata fields Extensions will inevitably want to extend the set of changeset data/fields that can be requested. We'll need to implement support for extending this in the future. Add some TODOs to track that. Differential Revision: https://phab.mercurial-scm.org/D4487

exchangev2: fetch and apply bookmarks This is pretty similar to phases data. We collect bookmarks data as we process records. Then at the end we make a call to the bookmarks subsystem to reflect the remote's bookmarks. Like phases, the code for handling bookmarks is vastly simpler than the previous wire protocol code because the server always transfers the full set of bookmarks when bookmarks are requested. We don't have to keep track of whether we requested bookmarks or not. Differential Revision: https://phab.mercurial-scm.org/D4486

wireprotov2: add bookmarks to "changesetdata" command Like we did for phases, we want to emit bookmarks data attached to each changeset. The approach here is very similar to phases: we emit bookmarks data inline with requested revision data. But we emit records for nodes that weren't requested as well so consumers have access to the full set of defined bookmarks. Differential Revision: https://phab.mercurial-scm.org/D4485

exchangev2: fetch and apply phases data Now that the server supports emitting phases data, we can request it and apply it on the client. Because we may receive phases-only updates from the server, we no longer conditionally perform the "changesetdata" command depending on whether there are revisions to fetch. In the previous wire protocol, this case would result in us falling back to performing "listkeys" commands to look up phases, bookmarks, etc data. But since "changesetdata" is smart enough to handle metadata only fetches, we can keep things consistent. It's worth noting that because of the unified approach to changeset data retrieval, phase handling code in wire proto v2 exchange is drastically simpler. Contrast with all the code in exchange.py dealing with all the variations for obtaining phases data. Differential Revision: https://phab.mercurial-scm.org/D4484

wireprotov2: add phases to "changesetdata" command This commit teaches the "changesetdata" wire protocol command to emit the phase state for each changeset. This is a different approach from existing phase transfer in a few ways. Previously, if there are no new revisions (or we're not using bundle2), we perform a "listkeys" request to retrieve phase heads. And when revision data is being transferred with bundle2, phases data is encoded in a standalone bundle2 part. In both cases, phases data is logically decoupled from the changeset data and is encountered/applied after changeset revision data is received. The new wire protocol purposefully tries to more tightly associate changeset metadata (phases, bookmarks, obsolescence markers, etc) with the changeset revision and index data itself, rather than have it live as a separate entity that must be fetched and processed separately. I reckon that one reason we didn't do this before was it was difficult to add new data types/fields without breaking existing consumers. By using CBOR maps to transfer changeset data and putting clients in control of what fields are requested / present in those maps, we can easily add additional changeset data while maintaining backwards compatibility. I believe this to be a superior approach to the problem. That being said, for performance reasons, we may need to resort to alternative mechanisms for transferring data like phases. But for now, I think giving the wire protocol the ability to transfer changeset metadata next to the changeset itself is a powerful feature because it is a raw, changeset-centric data API. And if you build simple APIs for accessing the fundamental units of repository data, you enable client-side experimentation (partial clone, etc). If it turns out that we need specialized APIs or mechanisms for transferring data like phases, we can build in those APIs later. For now, I'd like to see how far we can get on simple APIs. It's worth noting that when phase data is being requested, the server will also emit changeset records for nodes in the bases specified by the "noderange" argument. This is to ensure that phase-only updates for nodes the client has are available to the client, even if no new changesets will be transferred. Differential Revision: https://phab.mercurial-scm.org/D4483

exchangev2: fetch changeset revisions All Mercurial repository data is derived from changesets: you can't do anything unless you have changesets. Therefore, it makes sense for changesets to be the first piece of data that we transfer as part of pull. To do this, we call our new "changesetdata" command, requesting parents and revision data. This gives us all the data that a changegroup delta group would give us. We simply normalize this data into what addgroup() expects and call that API on the changelog to bulk insert revisions into the changelog. Code in this commit is heavily borrowed from changegroup.cg1unpacker.apply(). Differential Revision: https://phab.mercurial-scm.org/D4482

wireprotov2: define and implement "changesetdata" command This commit introduces the "changesetdata" wire protocol command. The role of the command is to expose data associated with changelog revisions, including the raw revision data itself. This command is the first piece of a new clone/pull strategy that is built on top of domain-specific commands for data retrieval. Instead of a monolithic "getbundle" command that transfers all of the things, we'll be introducing commands for fetching specific pieces of data. Since the changeset is the fundamental unit from which we derive pointers to other data (manifests, file nodes, etc), it makes sense to start reimplementing pull with this data. The command accepts as arguments a set of root and head revisions defining the changesets that should be fetched as well as an explicit list of nodes. By default, the command returns only the node values: the client must explicitly request additional fields be added to the response. Current supported fields are the list of parent nodes and the revision fulltext. My plan is to eventually add support for transferring other data associated with changesets, including phases, bookmarks, obsolescence markers, etc. Since the response format is CBOR, we'll be able to add this data into the response object relatively easily (it should be as simple as adding a key in a map). The documentation captures a number of TODO items. Some of these may require BC breaking changes. That's fine: wire protocol v2 is still highly experimental. Differential Revision: https://phab.mercurial-scm.org/D4481

exchangev2: start to implement pull with wire protocol v2 Wire protocol version 2 will take a substantially different approach to exchange than version 1 (at least as far as pulling is concerned). This commit establishes a new exchangev2 module for holding code related to exchange using wire protocol v2. I could have added things to the existing exchange module. But it is already quite big. And doing things inline isn't in question because the existing code is already littered with conditional code for various states of support for the existing wire protocol as it evolved over 10+ years. A new module gives us a chance to make a clean break. This approach does mean we'll end up writing some duplicate code. And there's a significant chance we'll miss functionality as code is ported. The plan is to eventually add #testcase's to existing tests so the new wire protocol is tested side-by-side with the existing one. This will hopefully tease out any features that weren't ported properly. But before we get there, we need to build up support for the new exchange methods. Our journey towards implementing a new exchange begins with pulling. And pulling begins with discovery. The discovery code added to exchangev2 is heavily drawn from the following functions: * exchange._pulldiscoverychangegroup * discovery.findcommonincoming For now, we build on top of existing discovery mechanisms. The new wire protocol should be capable of doing things more efficiently. But I'd rather defer on this problem. To foster the transition, we invent a fake capability on the HTTPv2 peer and have the main pull code in exchange.py call into exchangev2 when the new wire protocol is being used. Differential Revision: https://phab.mercurial-scm.org/D4480

httppeer: expose capabilities for each command This will help code using peers to sniff out exactly what servers support. Differential Revision: https://phab.mercurial-scm.org/D4436