internals: document compression negotiation As part of adding zstd support to all of the things, we'll need to teach the wire protocol to support non-zlib compression formats. This commit documents how we'll implement that. To understand how we arrived at this proposal, let's look at how things are done today. The wire protocol today doesn't have a unified format. Instead, there is a limited facility for differentiating replies as successful or not. And, each command essentially defines its own response format. A significant deficiency in the current protocol is the lack of payload framing over the SSH transport. In the HTTP transport, chunked transfer is used and the end of an HTTP response body (and the end of a Mercurial command response) can be identified by a 0 length chunk. This is how HTTP chunked transfer works. But in the SSH transport, there is no such framing, at least for certain responses (notably the response to "getbundle" requests). Clients can't simply read until end of stream because the socket is persistent and reused for multiple requests. Clients need to know when they've encountered the end of a request but there is nothing simple for them to key off of to detect this. So what happens is the client must decode the payload (as opposed to being dumb and forwarding frames/packets). This means the payload itself needs to support identifying end of stream. In some cases (bundle2), it also means the payload can encode "error" or "interrupt" events telling the client to e.g. abort processing. The lack of framing on the SSH transport and the transfer of its responsibilities to e.g. bundle2 is a massive layering violation and a wart on the protocol architecture. It needs to be fixed someday by inventing a proper framing protocol. So about compression. The client transport abstractions have a "_callcompressable()" API. This API is called to invoke a remote command that will send a compressible response. The response is essentially a "streaming" response (no framing data at the Mercurial layer) that is fed into a decompressor. On the HTTP transport, the decompressor is zlib and only zlib. There is currently no mechanism for the client to specify an alternate compression format. And, clients don't advertise what compression formats they support or ask the server to send a specific compression format. Instead, it is assumed that non-error responses to "compressible" commands are zlib compressed. On the SSH transport, there is no compression at the Mercurial protocol layer. Instead, compression must be handled by SSH itself (e.g. `ssh -C`) or within the payload data (e.g. bundle compression). For the HTTP transport, adding new compression formats is pretty straightforward. Once you know what decompressor to use, you can stream data into the decompressor until you reach a 0 size HTTP chunk, at which point you are at end of stream. So our wire protocol changes for the HTTP transport are pretty straightforward: the client and server advertise what compression formats they support and an appropriate compression format is chosen. We introduce a new HTTP media type to hold compressed payloads. The header of the payload defines the compression format being used. Whoever is on the receiving end can sniff the first few bytes route to an appropriate decompressor. Support for multiple compression formats is advertised on both server and client. The server advertises a "compression" capability saying which compression formats it supports and in what order they are preferred. Clients advertise their support for multiple compression formats and media types via the introduced "X-HgProto" request header. Strictly speaking, servers don't need to advertise which compression formats they support. But doing so allows clients to fail fast if they don't support any of the formats the server does. This is useful in situations like sending bundles, where the client may have to perform expensive computation before sending data to the server. Rather than simply advertise a list of supported compression formats, we introduce an additional "httpmediatype" server capability advertising which media types the server supports. This means servers are explicit about what formats they exchange. IMO, this is superior to inferring support from other capabilities (like "compression"). By advertising compression support on each request in the "X-HgProto" header and media type and direction at the server level, we are able to gradually transition existing commands/responses to the new media type and possibly compression. Contrast with the old world, where we only supported a single media type and the use of compression was built-in to the semantics of the command on both client and server. In the new world, if "application/mercurial-0.2" is supported, compression is supported. It's that simple. It's worth noting that we explicitly don't use "Accept," "Accept-Encoding," "Content-Encoding," or "Transfer-Encoding" for content negotiation and compression. People knowledgeable of the HTTP specifications will say that we should use these because that's what they are designed to be used for. They have a point and I sympathize with the argument. Earlier versions of this commit even defined supported media types in the "Accept" header. However, my years of experience rolling out services leveraging HTTP has taught me to not trust the HTTP layer, especially if you are going outside the normal spec (such as using a custom "Content-Encoding" value to represent zstd streams). I've seen load balancers, proxies, and other network devices do very bad and unexpected things to HTTP messages (like insisting zlib compressed content is decoded and then re-encoded at a different compression level or even stripping compression completely). I've found that the best way to avoid surprises when writing protocols on top of HTTP is to use HTTP as a dumb transport as much as possible to minimize the chances that an "intelligent" agent between endpoints will muck with your data. While the widespread use of TLS is mitigating many intermediate network agents interfering with HTTP, there are still problems at the edges, with e.g. the origin HTTP server needing to convert HTTP to and from WSGI and buggy or feature-lacking HTTP client implementations. I've found the best way to avoid these problems is to avoid using headers like "Content-Encoding" and to bake as much logic as possible into media types and HTTP message bodies. The protocol changes in this commit do rely on a custom HTTP request header and the "Content-Type" headers. But we used them before, so we shouldn't be increasing our exposure to "bad" HTTP agents. For the SSH transport, we can't easily implement content negotiation to determine compression formats because the SSH transport has no content negotiation capabilities today. And without a framing protocol, we don't know how much data to feed into a decompressor. So in order to implement compression support on the SSH transport, we'd need to invent a mechanism to represent content types and an outer framing protocol to stream data robustly. While I'm fully capable of doing that, it is a lot of work and not something that should be undertaken lightly. My opinion is that if we're going to change the SSH transport protocol, we should take a long hard look at implementing a grand unified protocol that attempts to address all the deficiencies with the existing protocol. While I want this to happen, that would be massive scope bloat standing in the way of zstd support. So, I've decided to take the easy solution: the SSH transport will not gain support for multiple compression formats. Keep in mind it doesn't support *any* compression today. So essentially nothing is changing on the SSH front.

     1

# factotum.py - Plan 9 factotum integration for Mercurial

     2

#

     3

# Copyright (C) 2012 Steven Stallion <sstallion@gmail.com>

     4

#

     5

# This program is free software; you can redistribute it and/or modify it

     6

# under the terms of the GNU General Public License as published by the

     7

# Free Software Foundation; either version 2 of the License, or (at your

     8

# option) any later version.

     9

#

    10

# This program is distributed in the hope that it will be useful, but

    11

# WITHOUT ANY WARRANTY; without even the implied warranty of

    12

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General

    13

# Public License for more details.

    14

#

    15

# You should have received a copy of the GNU General Public License along

    16

# with this program; if not, write to the Free Software Foundation, Inc.,

    17

# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

    18

    19

'''http authentication with factotum

    20

    21

This extension allows the factotum(4) facility on Plan 9 from Bell Labs

    22

platforms to provide authentication information for HTTP access. Configuration

    23

entries specified in the auth section as well as authentication information

    24

provided in the repository URL are fully supported. If no prefix is specified,

    25

a value of "*" will be assumed.

    26

    27

By default, keys are specified as::

    28

    29

  proto=pass service=hg prefix=<prefix> user=<username> !password=<password>

    30

    31

If the factotum extension is unable to read the required key, one will be

    32

requested interactively.

    33

    34

A configuration section is available to customize runtime behavior. By

    35

default, these entries are::

    36

    37

  [factotum]

    38

  executable = /bin/auth/factotum

    39

  mountpoint = /mnt/factotum

    40

  service = hg

    41

    42

The executable entry defines the full path to the factotum binary. The

    43

mountpoint entry defines the path to the factotum file service. Lastly, the

    44

service entry controls the service name used when reading keys.

    45

    46

'''

    47

    48

from __future__ import absolute_import

    49

    50

import os

    51

from mercurial.i18n import _

    52

from mercurial import (

    53

    error,

    54

    httpconnection,

    55

    url,

    56

    util,

    57

)

    58

    59

urlreq = util.urlreq

    60

passwordmgr = url.passwordmgr

    61

    62

ERRMAX = 128

    63

    64

_executable = _mountpoint = _service = None

    65

    66

def auth_getkey(self, params):

    67

    if not self.ui.interactive():

    68

        raise error.Abort(_('factotum not interactive'))

    69

    if 'user=' not in params:

    70

        params = '%s user?' % params

    71

    params = '%s !password?' % params

    72

    os.system("%s -g '%s'" % (_executable, params))

    73

    74

def auth_getuserpasswd(self, getkey, params):

    75

    params = 'proto=pass %s' % params

    76

    while True:

    77

        fd = os.open('%s/rpc' % _mountpoint, os.O_RDWR)

    78

        try:

    79

            os.write(fd, 'start %s' % params)

    80

            l = os.read(fd, ERRMAX).split()

    81

            if l[0] == 'ok':

    82

                os.write(fd, 'read')

    83

                status, user, passwd = os.read(fd, ERRMAX).split(None, 2)

    84

                if status == 'ok':

    85

                    if passwd.startswith("'"):

    86

                        if passwd.endswith("'"):

    87

                            passwd = passwd[1:-1].replace("''", "'")

    88

                        else:

    89

                            raise error.Abort(_('malformed password string'))

    90

                    return (user, passwd)

    91

        except (OSError, IOError):

    92

            raise error.Abort(_('factotum not responding'))

    93

        finally:

    94

            os.close(fd)

    95

        getkey(self, params)

    96

    97

def monkeypatch_method(cls):

    98

    def decorator(func):

    99

        setattr(cls, func.__name__, func)

   100

        return func

   101

    return decorator

   102

   103

@monkeypatch_method(passwordmgr)

   104

def find_user_password(self, realm, authuri):

   105

    user, passwd = self.passwddb.find_user_password(realm, authuri)

   106

    if user and passwd:

   107

        self._writedebug(user, passwd)

   108

        return (user, passwd)

   109

   110

    prefix = ''

   111

    res = httpconnection.readauthforuri(self.ui, authuri, user)

   112

    if res:

   113

        _, auth = res

   114

        prefix = auth.get('prefix')

   115

        user, passwd = auth.get('username'), auth.get('password')

   116

    if not user or not passwd:

   117

        if not prefix:

   118

            prefix = realm.split(' ')[0].lower()

   119

        params = 'service=%s prefix=%s' % (_service, prefix)

   120

        if user:

   121

            params = '%s user=%s' % (params, user)

   122

        user, passwd = auth_getuserpasswd(self, auth_getkey, params)

   123

   124

    self.add_password(realm, authuri, user, passwd)

   125

    self._writedebug(user, passwd)

   126

    return (user, passwd)

   127

   128

def uisetup(ui):

   129

    global _executable

   130

    _executable = ui.config('factotum', 'executable', '/bin/auth/factotum')

   131

    global _mountpoint

   132

    _mountpoint = ui.config('factotum', 'mountpoint', '/mnt/factotum')

   133

    global _service

   134

    _service = ui.config('factotum', 'service', 'hg')