hgext/clonebundles.py
author timeless <timeless@mozdev.org>
Tue, 24 Nov 2015 20:54:14 +0000
changeset 27099 e94bfebb4f35
parent 26884 762bf510b42c
child 27347 7807fe2795fb
permissions -rw-r--r--
contrib/perf: perfparents handle tiny repos refuse to run if there are not enough commits
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     1
# This software may be used and distributed according to the terms of the
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     2
# GNU General Public License version 2 or any later version.
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     3
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
     4
"""advertise pre-generated bundles to seed clones (experimental)
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
     5
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
     6
"clonebundles" is a server-side extension used to advertise the existence
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
     7
of pre-generated, externally hosted bundle files to clients that are
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
     8
cloning so that cloning can be faster, more reliable, and require less
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
     9
resources on the server.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    10
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    11
Cloning can be a CPU and I/O intensive operation on servers. Traditionally,
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    12
the server, in response to a client's request to clone, dynamically generates
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    13
a bundle containing the entire repository content and sends it to the client.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    14
There is no caching on the server and the server will have to redundantly
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    15
generate the same outgoing bundle in response to each clone request. For
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    16
servers with large repositories or with high clone volume, the load from
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    17
clones can make scaling the server challenging and costly.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    18
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    19
This extension provides server operators the ability to offload potentially
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    20
expensive clone load to an external service. Here's how it works.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    21
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    22
1. A server operator establishes a mechanism for making bundle files available
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    23
   on a hosting service where Mercurial clients can fetch them.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    24
2. A manifest file listing available bundle URLs and some optional metadata
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    25
   is added to the Mercurial repository on the server.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    26
3. A client initiates a clone against a clone bundles aware server.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    27
4. The client sees the server is advertising clone bundles and fetches the
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    28
   manifest listing available bundles.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    29
5. The client filters and sorts the available bundles based on what it
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    30
   supports and prefers.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    31
6. The client downloads and applies an available bundle from the
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    32
   server-specified URL.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    33
7. The client reconnects to the original server and performs the equivalent
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    34
   of :hg:`pull` to retrieve all repository data not in the bundle. (The
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    35
   repository could have been updated between when the bundle was created
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    36
   and when the client started the clone.)
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    37
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    38
Instead of the server generating full repository bundles for every clone
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    39
request, it generates full bundles once and they are subsequently reused to
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    40
bootstrap new clones. The server may still transfer data at clone time.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    41
However, this is only data that has been added/changed since the bundle was
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    42
created. For large, established repositories, this can reduce server load for
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    43
clones to less than 1% of original.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    44
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    45
To work, this extension requires the following of server operators:
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    46
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    47
* Generating bundle files of repository content (typically periodically,
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    48
  such as once per day).
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    49
* A file server that clients have network access to and that Python knows
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    50
  how to talk to through its normal URL handling facility (typically a
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    51
  HTTP server).
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    52
* A process for keeping the bundles manifest in sync with available bundle
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    53
  files.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    54
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    55
Strictly speaking, using a static file hosting server isn't required: a server
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    56
operator could use a dynamic service for retrieving bundle data. However,
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    57
static file hosting services are simple and scalable and should be sufficient
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    58
for most needs.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    59
26884
762bf510b42c clonebundles: fix typo s/comand/command/
Javi Merino <merino.jav@gmail.com>
parents: 26857
diff changeset
    60
Bundle files can be generated with the :hg:`bundle` command. Typically
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    61
:hg:`bundle --all` is used to produce a bundle of the entire repository.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    62
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    63
:hg:`debugcreatestreamclonebundle` can be used to produce a special
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    64
*streaming clone bundle*. These are bundle files that are extremely efficient
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    65
to produce and consume (read: fast). However, they are larger than
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    66
traditional bundle formats and require that clients support the exact set
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    67
of repository data store formats in use by the repository that created them.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    68
Typically, a newer server can serve data that is compatible with older clients.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    69
However, *streaming clone bundles* don't have this guarantee. **Server
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    70
operators need to be aware that newer versions of Mercurial may produce
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    71
streaming clone bundles incompatible with older Mercurial versions.**
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    72
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    73
The list of requirements printed by :hg:`debugcreatestreamclonebundle` should
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    74
be specified in the ``requirements`` parameter of the *bundle specification
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    75
string* for the ``BUNDLESPEC`` manifest property described below. e.g.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    76
``BUNDLESPEC=none-packed1;requirements%3Drevlogv1``.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    77
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    78
A server operator is responsible for creating a ``.hg/clonebundles.manifest``
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    79
file containing the list of available bundle files suitable for seeding
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    80
clones. If this file does not exist, the repository will not advertise the
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    81
existence of clone bundles when clients connect.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    82
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    83
The manifest file contains a newline (\n) delimited list of entries.
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    84
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    85
Each line in this file defines an available bundle. Lines have the format:
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    86
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    87
    <URL> [<key>=<value>[ <key>=<value>]]
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    88
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    89
That is, a URL followed by an optional, space-delimited list of key=value
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    90
pairs describing additional properties of this bundle. Both keys and values
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    91
are URI encoded.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    92
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    93
Keys in UPPERCASE are reserved for use by Mercurial and are defined below.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    94
All non-uppercase keys can be used by site installations. An example use
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    95
for custom properties is to use the *datacenter* attribute to define which
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    96
data center a file is hosted in. Clients could then prefer a server in the
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    97
data center closest to them.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    98
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
    99
The following reserved keys are currently defined:
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   100
26644
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   101
BUNDLESPEC
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   102
   A "bundle specification" string that describes the type of the bundle.
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   103
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   104
   These are string values that are accepted by the "--type" argument of
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   105
   :hg:`bundle`.
26644
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   106
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   107
   The values are parsed in strict mode, which means they must be of the
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   108
   "<compression>-<type>" form. See
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   109
   mercurial.exchange.parsebundlespec() for more details.
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   110
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   111
   Clients will automatically filter out specifications that are unknown or
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   112
   unsupported so they won't attempt to download something that likely won't
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   113
   apply.
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   114
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   115
   The actual value doesn't impact client behavior beyond filtering:
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   116
   clients will still sniff the bundle type from the header of downloaded
74de1c59f71c clonebundles: filter on bundle specification
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26623
diff changeset
   117
   files.
26645
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   118
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   119
   **Use of this key is highly recommended**, as it allows clients to
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   120
   easily skip unsupported bundles.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   121
26645
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   122
REQUIRESNI
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   123
   Whether Server Name Indication (SNI) is required to connect to the URL.
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   124
   SNI allows servers to use multiple certificates on the same IP. It is
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   125
   somewhat common in CDNs and other hosting providers. Older Python
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   126
   versions do not support SNI. Defining this attribute enables clients
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   127
   with older Python versions to filter this entry without experiencing
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   128
   an opaque SSL failure at connection time.
26645
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   129
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   130
   If this is defined, it is important to advertise a non-SNI fallback
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   131
   URL or clients running old Python releases may not be able to clone
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   132
   with the clonebundles facility.
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   133
2faa7671a4b3 clonebundles: filter on SNI requirement
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26644
diff changeset
   134
   Value should be "true".
26762
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   135
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   136
Manifests can contain multiple entries. Assuming metadata is defined, clients
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   137
will filter entries from the manifest that they don't support. The remaining
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   138
entries are optionally sorted by client preferences
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   139
(``experimental.clonebundleprefers`` config option). The client then attempts
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   140
to fetch the bundle at the first URL in the remaining list.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   141
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   142
**Errors when downloading a bundle will fail the entire clone operation:
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   143
clients do not automatically fall back to a traditional clone.** The reason
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   144
for this is that if a server is using clone bundles, it is probably doing so
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   145
because the feature is necessary to help it scale. In other words, there
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   146
is an assumption that clone load will be offloaded to another service and
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   147
that the Mercurial server isn't responsible for serving this clone load.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   148
If that other service experiences issues and clients start mass falling back to
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   149
the original Mercurial server, the added clone load could overwhelm the server
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   150
due to unexpected load and effectively take it offline. Not having clients
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   151
automatically fall back to cloning from the original server mitigates this
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   152
scenario.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   153
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   154
Because there is no automatic Mercurial server fallback on failure of the
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   155
bundle hosting service, it is important for server operators to view the bundle
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   156
hosting service as an extension of the Mercurial server in terms of
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   157
availability and service level agreements: if the bundle hosting service goes
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   158
down, so does the ability for clients to clone. Note: clients will see a
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   159
message informing them how to bypass the clone bundles facility when a failure
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   160
occurs. So server operators should prepare for some people to follow these
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   161
instructions when a failure occurs, thus driving more load to the original
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   162
Mercurial server when the bundle hosting service fails.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   163
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   164
The following config options influence the behavior of the clone bundles
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   165
feature:
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   166
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   167
ui.clonebundleadvertise
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   168
   Whether the server advertises the existence of the clone bundles feature
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   169
   to compatible clients that aren't using it.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   170
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   171
   When this is enabled (the default), a server will send a message to
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   172
   compatible clients performing a traditional clone informing them of the
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   173
   available clone bundles feature. Compatible clients are those that support
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   174
   bundle2 and are advertising support for the clone bundles feature.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   175
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   176
ui.clonebundlefallback
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   177
   Whether to automatically fall back to a traditional clone in case of
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   178
   clone bundles failure. Defaults to false for reasons described above.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   179
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   180
experimental.clonebundles
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   181
   Whether the clone bundles feature is enabled on clients. Defaults to true.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   182
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   183
experimental.clonebundleprefers
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   184
   List of "key=value" properties the client prefers in bundles. Downloaded
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   185
   bundle manifests will be sorted by the preferences in this list. e.g.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   186
   the value "BUNDLESPEC=gzip-v1, BUNDLESPEC=bzip2=v1" will prefer a gzipped
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   187
   version 1 bundle type then bzip2 version 1 bundle type.
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   188
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   189
   If not defined, the order in the manifest will be used and the first
26f622859288 clonebundles: rewrite documentation
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26691
diff changeset
   190
   available bundle will be downloaded.
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   191
"""
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   192
26691
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   193
from mercurial.i18n import _
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   194
from mercurial.node import nullid
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   195
from mercurial import (
26691
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   196
    exchange,
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   197
    extensions,
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   198
    wireproto,
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   199
)
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   200
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   201
testedwith = 'internal'
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   202
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   203
def capabilities(orig, repo, proto):
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   204
    caps = orig(repo, proto)
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   205
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   206
    # Only advertise if a manifest exists. This does add some I/O to requests.
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   207
    # But this should be cheaper than a wasted network round trip due to
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   208
    # missing file.
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   209
    if repo.opener.exists('clonebundles.manifest'):
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   210
        caps.append('clonebundles')
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   211
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   212
    return caps
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   213
26691
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   214
@exchange.getbundle2partsgenerator('clonebundlesadvertise', 0)
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   215
def advertiseclonebundlespart(bundler, repo, source, bundlecaps=None,
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   216
                              b2caps=None, heads=None, common=None,
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   217
                              cbattempted=None, **kwargs):
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   218
    """Inserts an output part to advertise clone bundles availability."""
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   219
    # Allow server operators to disable this behavior.
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   220
    # # experimental config: ui.clonebundleadvertise
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   221
    if not repo.ui.configbool('ui', 'clonebundleadvertise', True):
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   222
        return
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   223
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   224
    # Only advertise if a manifest is present.
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   225
    if not repo.opener.exists('clonebundles.manifest'):
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   226
        return
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   227
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   228
    # And when changegroup data is requested.
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   229
    if not kwargs.get('cg', True):
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   230
        return
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   231
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   232
    # And when the client supports clone bundles.
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   233
    if cbattempted is None:
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   234
        return
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   235
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   236
    # And when the client didn't attempt a clone bundle as part of this pull.
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   237
    if cbattempted:
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   238
        return
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   239
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   240
    # And when a full clone is requested.
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   241
    # Note: client should not send "cbattempted" for regular pulls. This check
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   242
    # is defense in depth.
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   243
    if common and common != [nullid]:
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   244
        return
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   245
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   246
    msg = _('this server supports the experimental "clone bundles" feature '
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   247
            'that should enable faster and more reliable cloning\n'
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   248
            'help test it by setting the "experimental.clonebundles" config '
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   249
            'flag to "true"')
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   250
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   251
    bundler.newpart('output', data=msg)
23c0da28c034 clonebundles: advertise clone bundles feature to clients
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26645
diff changeset
   252
26623
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   253
def extsetup(ui):
5a95fe44121d clonebundles: support for seeding clones from pre-generated bundles
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   254
    extensions.wrapfunction(wireproto, '_capabilities', capabilities)