The Mercurial wire protocol is a request-response based protocol

with multiple wire representations.

Each request is modeled as a command name, a dictionary of arguments, and

optional raw input. Command arguments and their types are intrinsic

properties of commands. So is the response type of the command. This means

clients can't always send arbitrary arguments to servers and servers can't

return multiple response types.

The protocol is synchronous and does not support multiplexing (concurrent

commands).

Handshake

=========

It is required or common for clients to perform a *handshake* when connecting

to a server. The handshake serves the following purposes:

* Negotiating protocol/transport level options

* Allows the client to learn about server capabilities to influence

  future requests

* Ensures the underlying transport channel is in a *clean* state

An important goal of the handshake is to allow clients to use more modern

wire protocol features. By default, clients must assume they are talking

to an old version of Mercurial server (possibly even the very first

implementation). So, clients should not attempt to call or utilize modern

wire protocol features until they have confirmation that the server

supports them. The handshake implementation is designed to allow both

ends to utilize the latest set of features and capabilities with as

few round trips as possible.

The handshake mechanism varies by transport and protocol and is documented

in the sections below.

HTTP Protocol

=============

Handshake

---------

The client sends a ``capabilities`` command request (``?cmd=capabilities``)

as soon as HTTP requests may be issued.

The server responds with a capabilities string, which the client parses to

learn about the server's abilities.

HTTP Version 1 Transport

------------------------

Commands are issued as HTTP/1.0 or HTTP/1.1 requests. Commands are

sent to the base URL of the repository with the command name sent in

the ``cmd`` query string parameter. e.g.

``https://example.com/repo?cmd=capabilities``. The HTTP method is ``GET``

or ``POST`` depending on the command and whether there is a request

body.

Command arguments can be sent multiple ways.

The simplest is part of the URL query string using ``x-www-form-urlencoded``

encoding (see Python's ``urllib.urlencode()``. However, many servers impose

length limitations on the URL. So this mechanism is typically only used if

the server doesn't support other mechanisms.

If the server supports the ``httpheader`` capability, command arguments can

be sent in HTTP request headers named ``X-HgArg-<N>`` where ``<N>`` is an

integer starting at 1. A ``x-www-form-urlencoded`` representation of the

arguments is obtained. This full string is then split into chunks and sent

in numbered ``X-HgArg-<N>`` headers. The maximum length of each HTTP header

is defined by the server in the ``httpheader`` capability value, which defaults

to ``1024``. The server reassembles the encoded arguments string by

concatenating the ``X-HgArg-<N>`` headers then URL decodes them into a

dictionary.

The list of ``X-HgArg-<N>`` headers should be added to the ``Vary`` request

header to instruct caches to take these headers into consideration when caching

requests.

If the server supports the ``httppostargs`` capability, the client

may send command arguments in the HTTP request body as part of an

HTTP POST request. The command arguments will be URL encoded just like

they would for sending them via HTTP headers. However, no splitting is

performed: the raw arguments are included in the HTTP request body.

The client sends a ``X-HgArgs-Post`` header with the string length of the

encoded arguments data. Additional data may be included in the HTTP

request body immediately following the argument data. The offset of the

non-argument data is defined by the ``X-HgArgs-Post`` header. The

``X-HgArgs-Post`` header is not required if there is no argument data.

Additional command data can be sent as part of the HTTP request body. The

default ``Content-Type`` when sending data is ``application/mercurial-0.1``.

A ``Content-Length`` header is currently always sent.

Example HTTP requests::

    GET /repo?cmd=capabilities

    X-HgArg-1: foo=bar&baz=hello%20world

The request media type should be chosen based on server support. If the

``httpmediatype`` server capability is present, the client should send

the newest mutually supported media type. If this capability is absent,

the client must assume the server only supports the

``application/mercurial-0.1`` media type.

The ``Content-Type`` HTTP response header identifies the response as coming

from Mercurial and can also be used to signal an error has occurred.

The ``application/mercurial-*`` media types indicate a generic Mercurial

data type.

The ``application/mercurial-0.1`` media type is raw Mercurial data. It is the

predecessor of the format below.

The ``application/mercurial-0.2`` media type is compression framed Mercurial

data. The first byte of the payload indicates the length of the compression

format identifier that follows. Next are N bytes indicating the compression

format. e.g. ``zlib``. The remaining bytes are compressed according to that

compression format. The decompressed data behaves the same as with

``application/mercurial-0.1``.

The ``application/hg-error`` media type indicates a generic error occurred.

The content of the HTTP response body typically holds text describing the

error.

The ``application/hg-changegroup`` media type indicates a changegroup response

type.

Clients also accept the ``text/plain`` media type. All other media

types should cause the client to error.

Behavior of media types is further described in the ``Content Negotiation``

section below.

Clients should issue a ``User-Agent`` request header that identifies the client.

The server should not use the ``User-Agent`` for feature detection.

A command returning a ``string`` response issues a

``application/mercurial-0.*`` media type and the HTTP response body contains

the raw string value (after compression decoding, if used). A

``Content-Length`` header is typically issued, but not required.

A command returning a ``stream`` response issues a

``application/mercurial-0.*`` media type and the HTTP response is typically

using *chunked transfer* (``Transfer-Encoding: chunked``).

HTTP Version 2 Transport

------------------------

**Experimental - feature under active development**

Version 2 of the HTTP protocol is exposed under the ``/api/*`` URL space.

It's final API name is not yet formalized.

Commands are triggered by sending HTTP POST requests against URLs of the

form ``<permission>/<command>``, where ``<permission>`` is ``ro`` or

``rw``, meaning read-only and read-write, respectively and ``<command>``

is a named wire protocol command.

Non-POST request methods MUST be rejected by the server with an HTTP

405 response.

Commands that modify repository state in meaningful ways MUST NOT be

exposed under the ``ro`` URL prefix. All available commands MUST be

available under the ``rw`` URL prefix.

Server adminstrators MAY implement blanket HTTP authentication keyed

off the URL prefix. For example, a server may require authentication

for all ``rw/*`` URLs and let unauthenticated requests to ``ro/*``

URL proceed. A server MAY issue an HTTP 401, 403, or 407 response

in accordance with RFC 7235. Clients SHOULD recognize the HTTP Basic

(RFC 7617) and Digest (RFC 7616) authentication schemes. Clients SHOULD

make an attempt to recognize unknown schemes using the

``WWW-Authenticate`` response header on a 401 response, as defined by

RFC 7235.

Read-only commands are accessible under ``rw/*`` URLs so clients can

signal the intent of the operation very early in the connection

lifecycle. For example, a ``push`` operation - which consists of

various read-only commands mixed with at least one read-write command -

can perform all commands against ``rw/*`` URLs so that any server-side

authentication requirements are discovered upon attempting the first

command - not potentially several commands into the exchange. This

allows clients to fail faster or prompt for credentials as soon as the

exchange takes place. This provides a better end-user experience.

Requests to unknown commands or URLS result in an HTTP 404.

TODO formally define response type, how error is communicated, etc.

HTTP request and response bodies use the *Unified Frame-Based Protocol*

(defined below) for media exchange. The entirety of the HTTP message

body is 0 or more frames as defined by this protocol.

Clients and servers MUST advertise the ``TBD`` media type via the

``Content-Type`` request and response headers. In addition, clients MUST

advertise this media type value in their ``Accept`` request header in all

requests.

TODO finalize the media type. For now, it is defined in wireprotoserver.py.

Servers receiving requests without an ``Accept`` header SHOULD respond with

an HTTP 406.

Servers receiving requests with an invalid ``Content-Type`` header SHOULD

respond with an HTTP 415.

The command to run is specified in the POST payload as defined by the

*Unified Frame-Based Protocol*. This is redundant with data already

encoded in the URL. This is by design, so server operators can have

better understanding about server activity from looking merely at

HTTP access logs.

In most circumstances, the command specified in the URL MUST match

the command specified in the frame-based payload or the server will

respond with an error. The exception to this is the special

``multirequest`` URL. (See below.) In addition, HTTP requests

are limited to one command invocation. The exception is the special

``multirequest`` URL.

The ``multirequest`` command endpoints (``ro/multirequest`` and

``rw/multirequest``) are special in that they allow the execution of

*any* command and allow the execution of multiple commands. If the

HTTP request issues multiple commands across multiple frames, all

issued commands will be processed by the server. Per the defined

behavior of the *Unified Frame-Based Protocol*, commands may be

issued interleaved and responses may come back in a different order

than they were issued. Clients MUST be able to deal with this.

SSH Protocol

============

Handshake

---------

For all clients, the handshake consists of the client sending 1 or more

commands to the server using version 1 of the transport. Servers respond

to commands they know how to respond to and send an empty response (``0\n``)

for unknown commands (per standard behavior of version 1 of the transport).

Clients then typically look for a response to the newest sent command to

determine which transport version to use and what the available features for

the connection and server are.

Preceding any response from client-issued commands, the server may print

non-protocol output. It is common for SSH servers to print banners, message

of the day announcements, etc when clients connect. It is assumed that any

such *banner* output will precede any Mercurial server output. So clients

must be prepared to handle server output on initial connect that isn't

in response to any client-issued command and doesn't conform to Mercurial's

wire protocol. This *banner* output should only be on stdout. However,

some servers may send output on stderr.

Pre 0.9.1 clients issue a ``between`` command with the ``pairs`` argument

having the value

``0000000000000000000000000000000000000000-0000000000000000000000000000000000000000``.

The ``between`` command has been supported since the original Mercurial

SSH server. Requesting the empty range will return a ``\n`` string response,

which will be encoded as ``1\n\n`` (value length of ``1`` followed by a newline

followed by the value, which happens to be a newline).

For pre 0.9.1 clients and all servers, the exchange looks like::

   c: between\n

   c: pairs 81\n

   c: 0000000000000000000000000000000000000000-0000000000000000000000000000000000000000

   s: 1\n

   s: \n

0.9.1+ clients send a ``hello`` command (with no arguments) before the

``between`` command. The response to this command allows clients to

discover server capabilities and settings.

An example exchange between 0.9.1+ clients and a ``hello`` aware server looks

like::

   c: hello\n

   c: between\n

   c: pairs 81\n

   c: 0000000000000000000000000000000000000000-0000000000000000000000000000000000000000

   s: 324\n

   s: capabilities: lookup changegroupsubset branchmap pushkey known getbundle ...\n

   s: 1\n

   s: \n

And a similar scenario but with servers sending a banner on connect::

   c: hello\n

   c: between\n

   c: pairs 81\n

   c: 0000000000000000000000000000000000000000-0000000000000000000000000000000000000000

   s: welcome to the server\n

   s: if you find any issues, email someone@somewhere.com\n

   s: 324\n

   s: capabilities: lookup changegroupsubset branchmap pushkey known getbundle ...\n

   s: 1\n

   s: \n

Note that output from the ``hello`` command is terminated by a ``\n``. This is

part of the response payload and not part of the wire protocol adding a newline

after responses. In other words, the length of the response contains the

trailing ``\n``.

Clients supporting version 2 of the SSH transport send a line beginning

with ``upgrade`` before the ``hello`` and ``between`` commands. The line

(which isn't a well-formed command line because it doesn't consist of a

single command name) serves to both communicate the client's intent to

switch to transport version 2 (transports are version 1 by default) as

well as to advertise the client's transport-level capabilities so the

server may satisfy that request immediately.

The upgrade line has the form:

    upgrade <token> <transport capabilities>

That is the literal string ``upgrade`` followed by a space, followed by

a randomly generated string, followed by a space, followed by a string

denoting the client's transport capabilities.

The token can be anything. However, a random UUID is recommended. (Use

of version 4 UUIDs is recommended because version 1 UUIDs can leak the

client's MAC address.)

The transport capabilities string is a URL/percent encoded string

containing key-value pairs defining the client's transport-level

capabilities. The following capabilities are defined:

proto

   A comma-delimited list of transport protocol versions the client

   supports. e.g. ``ssh-v2``.

If the server does not recognize the ``upgrade`` line, it should issue

an empty response and continue processing the ``hello`` and ``between``

commands. Here is an example handshake between a version 2 aware client

and a non version 2 aware server:

   c: upgrade 2e82ab3f-9ce3-4b4e-8f8c-6fd1c0e9e23a proto=ssh-v2

   c: hello\n

   c: between\n

   c: pairs 81\n

   c: 0000000000000000000000000000000000000000-0000000000000000000000000000000000000000

   s: 0\n

   s: 324\n

   s: capabilities: lookup changegroupsubset branchmap pushkey known getbundle ...\n

   s: 1\n

   s: \n

(The initial ``0\n`` line from the server indicates an empty response to

the unknown ``upgrade ..`` command/line.)

If the server recognizes the ``upgrade`` line and is willing to satisfy that

upgrade request, it replies to with a payload of the following form:

   upgraded <token> <transport name>\n

This line is the literal string ``upgraded``, a space, the token that was

specified by the client in its ``upgrade ...`` request line, a space, and the

name of the transport protocol that was chosen by the server. The transport

name MUST match one of the names the client specified in the ``proto`` field

of its ``upgrade ...`` request line.

If a server issues an ``upgraded`` response, it MUST also read and ignore

the lines associated with the ``hello`` and ``between`` command requests

that were issued by the server. It is assumed that the negotiated transport

will respond with equivalent requested information following the transport

handshake.

All data following the ``\n`` terminating the ``upgraded`` line is the

domain of the negotiated transport. It is common for the data immediately

following to contain additional metadata about the state of the transport and

the server. However, this isn't strictly speaking part of the transport

handshake and isn't covered by this section.

Here is an example handshake between a version 2 aware client and a version

2 aware server:

   c:  upgrade 2e82ab3f-9ce3-4b4e-8f8c-6fd1c0e9e23a proto=ssh-v2

   c:  hello\n

   c:  between\n

   c:  pairs 81\n

   c:  0000000000000000000000000000000000000000-0000000000000000000000000000000000000000

   s: upgraded 2e82ab3f-9ce3-4b4e-8f8c-6fd1c0e9e23a ssh-v2\n

   s: <additional transport specific data>

The client-issued token that is echoed in the response provides a more

resilient mechanism for differentiating *banner* output from Mercurial

output. In version 1, properly formatted banner output could get confused

for Mercurial server output. By submitting a randomly generated token

that is then present in the response, the client can look for that token

in response lines and have reasonable certainty that the line did not

originate from a *banner* message.

SSH Version 1 Transport

-----------------------

The SSH transport (version 1) is a custom text-based protocol suitable for

use over any bi-directional stream transport. It is most commonly used with

SSH.

A SSH transport server can be started with ``hg serve --stdio``. The stdin,

stderr, and stdout file descriptors of the started process are used to exchange

data. When Mercurial connects to a remote server over SSH, it actually starts

a ``hg serve --stdio`` process on the remote server.

Commands are issued by sending the command name followed by a trailing newline

``\n`` to the server. e.g. ``capabilities\n``.

Command arguments are sent in the following format::

    <argument> <length>\n<value>

That is, the argument string name followed by a space followed by the

integer length of the value (expressed as a string) followed by a newline

(``\n``) followed by the raw argument value.

Dictionary arguments are encoded differently::

    <argument> <# elements>\n

    <key1> <length1>\n<value1>

    <key2> <length2>\n<value2>

    ...

Non-argument data is sent immediately after the final argument value. It is

encoded in chunks::

    <length>\n<data>

Each command declares a list of supported arguments and their types. If a

client sends an unknown argument to the server, the server should abort

immediately. The special argument ``*`` in a command's definition indicates

that all argument names are allowed.

The definition of supported arguments and types is initially made when a

new command is implemented. The client and server must initially independently

agree on the arguments and their types. This initial set of arguments can be

supplemented through the presence of *capabilities* advertised by the server.

Each command has a defined expected response type.

A ``string`` response type is a length framed value. The response consists of

the string encoded integer length of a value followed by a newline (``\n``)

followed by the value. Empty values are allowed (and are represented as

``0\n``).

A ``stream`` response type consists of raw bytes of data. There is no framing.

A generic error response type is also supported. It consists of a an error

message written to ``stderr`` followed by ``\n-\n``. In addition, ``\n`` is

written to ``stdout``.

If the server receives an unknown command, it will send an empty ``string``

response.

The server terminates if it receives an empty command (a ``\n`` character).

If the server announces support for the ``protocaps`` capability, the client

should issue a ``protocaps`` command after the initial handshake to annonunce

its own capabilities. The client capabilities are persistent.

SSH Version 2 Transport

-----------------------

**Experimental and under development**

Version 2 of the SSH transport behaves identically to version 1 of the SSH

transport with the exception of handshake semantics. See above for how

version 2 of the SSH transport is negotiated.

Immediately following the ``upgraded`` line signaling a switch to version

2 of the SSH protocol, the server automatically sends additional details

about the capabilities of the remote server. This has the form:

   <integer length of value>\n

   capabilities: ...\n

e.g.

   s: upgraded 2e82ab3f-9ce3-4b4e-8f8c-6fd1c0e9e23a ssh-v2\n

   s: 240\n

   s: capabilities: known getbundle batch ...\n

Following capabilities advertisement, the peers communicate using version

1 of the SSH transport.

Unified Frame-Based Protocol

============================

**Experimental and under development**

The *Unified Frame-Based Protocol* is a communications protocol between

Mercurial peers. The protocol aims to be mostly transport agnostic

(works similarly on HTTP, SSH, etc).

To operate the protocol, a bi-directional, half-duplex pipe supporting

ordered sends and receives is required. That is, each peer has one pipe

for sending data and another for receiving.