mercurial/changegroup.py
author Matt Mackall <mpm@selenic.com>
Wed, 13 Dec 2006 13:27:09 -0600
changeset 3877 abaee83ce0a6
parent 3859 8c24b6fd5866
child 3891 6b4127c7d52a
permissions -rw-r--r--
Replace demandload with new demandimport
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     1
"""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     2
changegroup.py - Mercurial changegroup manipulation functions
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     3
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     4
 Copyright 2006 Matt Mackall <mpm@selenic.com>
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     5
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     6
This software may be used and distributed according to the terms
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     7
of the GNU General Public License, incorporated herein by reference.
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     8
"""
3877
abaee83ce0a6 Replace demandload with new demandimport
Matt Mackall <mpm@selenic.com>
parents: 3859
diff changeset
     9
1997
802e8a029d99 Added missing gettext import to changegroup.py.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 1981
diff changeset
    10
from i18n import gettext as _
3877
abaee83ce0a6 Replace demandload with new demandimport
Matt Mackall <mpm@selenic.com>
parents: 3859
diff changeset
    11
import struct, os, bz2, zlib, util, tempfile
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    12
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    13
def getchunk(source):
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    14
    """get a chunk from a changegroup"""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    15
    d = source.read(4)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    16
    if not d:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    17
        return ""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    18
    l = struct.unpack(">l", d)[0]
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    19
    if l <= 4:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    20
        return ""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    21
    d = source.read(l - 4)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    22
    if len(d) < l - 4:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    23
        raise util.Abort(_("premature EOF reading chunk"
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    24
                           " (got %d bytes, expected %d)")
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    25
                          % (len(d), l - 4))
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    26
    return d
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    27
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    28
def chunkiter(source):
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    29
    """iterate through the chunks in source"""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    30
    while 1:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    31
        c = getchunk(source)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    32
        if not c:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    33
            break
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    34
        yield c
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    35
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    36
def genchunk(data):
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    37
    """build a changegroup chunk"""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    38
    header = struct.pack(">l", len(data)+ 4)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    39
    return "%s%s" % (header, data)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    40
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    41
def closechunk():
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    42
    return struct.pack(">l", 0)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    43
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    44
class nocompress(object):
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    45
    def compress(self, x):
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    46
        return x
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    47
    def flush(self):
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    48
        return ""
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    49
3662
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3660
diff changeset
    50
bundletypes = {
3704
9c1737a3e254 fix writebundle for bz2 bundles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3662
diff changeset
    51
    "": ("", nocompress),
9c1737a3e254 fix writebundle for bz2 bundles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3662
diff changeset
    52
    "HG10UN": ("HG10UN", nocompress),
3762
b9d3e12da485 changegroup.py: delay the loading of the bz2 and zlib modules
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3706
diff changeset
    53
    "HG10BZ": ("HG10", lambda: bz2.BZ2Compressor()),
b9d3e12da485 changegroup.py: delay the loading of the bz2 and zlib modules
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3706
diff changeset
    54
    "HG10GZ": ("HG10GZ", lambda: zlib.compressobj()),
3662
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3660
diff changeset
    55
}
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3660
diff changeset
    56
3706
0d810798acb1 Use 'bundletype' instead of 'type' to not shadow built-in function.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 3705
diff changeset
    57
def writebundle(cg, filename, bundletype):
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    58
    """Write a bundle file and return its filename.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    59
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    60
    Existing files will not be overwritten.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    61
    If no filename is specified, a temporary file is created.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    62
    bz2 compression can be turned off.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    63
    The bundle file will be deleted in case of errors.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    64
    """
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    65
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    66
    fh = None
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    67
    cleanup = None
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    68
    try:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    69
        if filename:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    70
            if os.path.exists(filename):
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    71
                raise util.Abort(_("file '%s' already exists") % filename)
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    72
            fh = open(filename, "wb")
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    73
        else:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    74
            fd, filename = tempfile.mkstemp(prefix="hg-bundle-", suffix=".hg")
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    75
            fh = os.fdopen(fd, "wb")
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    76
        cleanup = filename
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    77
3706
0d810798acb1 Use 'bundletype' instead of 'type' to not shadow built-in function.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 3705
diff changeset
    78
        header, compressor = bundletypes[bundletype]
3704
9c1737a3e254 fix writebundle for bz2 bundles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3662
diff changeset
    79
        fh.write(header)
9c1737a3e254 fix writebundle for bz2 bundles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3662
diff changeset
    80
        z = compressor()
3662
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3660
diff changeset
    81
3659
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    82
        # parse the changegroup data, otherwise we will block
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    83
        # in case of sshrepo because we don't know the end of the stream
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    84
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    85
        # an empty chunkiter is the end of the changegroup
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    86
        empty = False
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    87
        while not empty:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    88
            empty = True
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    89
            for chunk in chunkiter(cg):
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    90
                empty = False
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    91
                fh.write(z.compress(genchunk(chunk)))
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    92
            fh.write(z.compress(closechunk()))
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    93
        fh.write(z.flush())
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    94
        cleanup = None
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    95
        return filename
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    96
    finally:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    97
        if fh is not None:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    98
            fh.close()
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    99
        if cleanup is not None:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   100
            os.unlink(cleanup)
3660
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   101
3859
8c24b6fd5866 fix errors spotted by pychecker
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3762
diff changeset
   102
def readbundle(fh, fname):
3660
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   103
    header = fh.read(6)
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   104
    if not header.startswith("HG"):
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   105
        raise util.Abort(_("%s: not a Mercurial bundle file") % fname)
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   106
    elif not header.startswith("HG10"):
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   107
        raise util.Abort(_("%s: unknown bundle version") % fname)
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   108
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   109
    if header == "HG10BZ":
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   110
        def generator(f):
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   111
            zd = bz2.BZ2Decompressor()
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   112
            zd.decompress("BZ")
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   113
            for chunk in util.filechunkiter(f, 4096):
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   114
                yield zd.decompress(chunk)
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   115
        return util.chunkbuffer(generator(fh))
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   116
    elif header == "HG10UN":
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   117
        return fh
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   118
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   119
    raise util.Abort(_("%s: unknown bundle compression type")
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3659
diff changeset
   120
                     % fname)