contrib/shrink-revlog.py
author Benoit Boissinot <benoit.boissinot@ens-lyon.org>
Wed, 24 Feb 2010 18:22:45 +0100
changeset 10542 989b2a5eaaba
parent 10509 3e7e789d9494
child 10620 1ee14abe07b4
child 11267 d3ebb1a0bc49
permissions -rwxr-xr-x
shrink: handle all combinations of inline/non-inline revlogs
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
     1
#!/usr/bin/env python
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
     2
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
     3
"""\
10236
49a8625b8cac shrink-revlog: help/doc tweaks
Greg Ward <greg-hg@gerg.ca>
parents: 10234
diff changeset
     4
reorder a revlog (the manifest by default) to save space
49a8625b8cac shrink-revlog: help/doc tweaks
Greg Ward <greg-hg@gerg.ca>
parents: 10234
diff changeset
     5
49a8625b8cac shrink-revlog: help/doc tweaks
Greg Ward <greg-hg@gerg.ca>
parents: 10234
diff changeset
     6
Specifically, this topologically sorts the revisions in the revlog so that
49a8625b8cac shrink-revlog: help/doc tweaks
Greg Ward <greg-hg@gerg.ca>
parents: 10234
diff changeset
     7
revisions on the same branch are adjacent as much as possible. This is a
49a8625b8cac shrink-revlog: help/doc tweaks
Greg Ward <greg-hg@gerg.ca>
parents: 10234
diff changeset
     8
workaround for the fact that Mercurial computes deltas relative to the
10216
843f6ee6d14b contrib: small documentation fixes in shrink-revlog.py
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10215
diff changeset
     9
previous revision rather than relative to a parent revision.
843f6ee6d14b contrib: small documentation fixes in shrink-revlog.py
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10215
diff changeset
    10
843f6ee6d14b contrib: small documentation fixes in shrink-revlog.py
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10215
diff changeset
    11
This is *not* safe to run on a changelog.
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    12
"""
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    13
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    14
# Originally written by Benoit Boissinot <benoit.boissinot at ens-lyon.org>
10216
843f6ee6d14b contrib: small documentation fixes in shrink-revlog.py
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10215
diff changeset
    15
# as a patch to rewrite-log. Cleaned up, refactored, documented, and
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    16
# renamed by Greg Ward <greg at gerg.ca>.
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    17
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    18
# XXX would be nice to have a way to verify the repository after shrinking,
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    19
# e.g. by comparing "before" and "after" states of random changesets
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    20
# (maybe: export before, shrink, export after, diff).
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    21
10542
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
    22
import os, tempfile, errno
10509
3e7e789d9494 shrink-revlog: remove unneeded imports and useless code
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10508
diff changeset
    23
from mercurial import revlog, transaction, node, util
10009
69dca8574a6a shrink-revlog: improve performance: use changegroup instead of revisions
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 9712
diff changeset
    24
from mercurial import changegroup
10508
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
    25
from mercurial.i18n import _
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    26
10213
9e6848f352b0 contrib: use ui to write in shrink-revlog.py
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10009
diff changeset
    27
def toposort(ui, rl):
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    28
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    29
    children = {}
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    30
    root = []
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    31
    # build children and roots
10508
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
    32
    ui.status(_('reading revs\n'))
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    33
    try:
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    34
        for i in rl:
10496
45734b51c99b progress: mark strings for translation
Martin Geisler <mg@lazybytes.net>
parents: 10440
diff changeset
    35
            ui.progress(_('reading'), i, total=len(rl))
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    36
            children[i] = []
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    37
            parents = [p for p in rl.parentrevs(i) if p != node.nullrev]
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    38
            # in case of duplicate parents
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    39
            if len(parents) == 2 and parents[0] == parents[1]:
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    40
                del parents[1]
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    41
            for p in parents:
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    42
                assert p in children
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    43
                children[p].append(i)
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    44
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    45
            if len(parents) == 0:
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    46
                root.append(i)
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    47
    finally:
10496
45734b51c99b progress: mark strings for translation
Martin Geisler <mg@lazybytes.net>
parents: 10440
diff changeset
    48
        ui.progress(_('reading'), None, total=len(rl))
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    49
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    50
    # XXX this is a reimplementation of the 'branchsort' topo sort
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    51
    # algorithm in hgext.convert.convcmd... would be nice not to duplicate
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    52
    # the algorithm
10508
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
    53
    ui.status(_('sorting revs\n'))
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    54
    visit = root
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    55
    ret = []
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    56
    while visit:
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    57
        i = visit.pop(0)
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    58
        ret.append(i)
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    59
        if i not in children:
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    60
            # This only happens if some node's p1 == p2, which can
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    61
            # happen in the manifest in certain circumstances.
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    62
            continue
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    63
        next = []
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    64
        for c in children.pop(i):
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    65
            parents_unseen = [p for p in rl.parentrevs(c)
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    66
                              if p != node.nullrev and p in children]
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    67
            if len(parents_unseen) == 0:
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    68
                next.append(c)
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    69
        visit = next + visit
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    70
    return ret
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    71
10213
9e6848f352b0 contrib: use ui to write in shrink-revlog.py
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10009
diff changeset
    72
def writerevs(ui, r1, r2, order, tr):
10009
69dca8574a6a shrink-revlog: improve performance: use changegroup instead of revisions
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 9712
diff changeset
    73
10508
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
    74
    ui.status(_('writing revs\n'))
10440
b39b32c33269 shrink: use progress API
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10282
diff changeset
    75
10009
69dca8574a6a shrink-revlog: improve performance: use changegroup instead of revisions
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 9712
diff changeset
    76
    count = [0]
69dca8574a6a shrink-revlog: improve performance: use changegroup instead of revisions
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 9712
diff changeset
    77
    def progress(*args):
10496
45734b51c99b progress: mark strings for translation
Martin Geisler <mg@lazybytes.net>
parents: 10440
diff changeset
    78
        ui.progress(_('writing'), count[0], total=len(order))
10009
69dca8574a6a shrink-revlog: improve performance: use changegroup instead of revisions
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 9712
diff changeset
    79
        count[0] += 1
69dca8574a6a shrink-revlog: improve performance: use changegroup instead of revisions
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 9712
diff changeset
    80
69dca8574a6a shrink-revlog: improve performance: use changegroup instead of revisions
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 9712
diff changeset
    81
    order = [r1.node(r) for r in order]
69dca8574a6a shrink-revlog: improve performance: use changegroup instead of revisions
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 9712
diff changeset
    82
69dca8574a6a shrink-revlog: improve performance: use changegroup instead of revisions
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 9712
diff changeset
    83
    # this is a bit ugly, but it works
69dca8574a6a shrink-revlog: improve performance: use changegroup instead of revisions
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 9712
diff changeset
    84
    lookup = lambda x: "%020d" % r1.linkrev(r1.rev(x))
69dca8574a6a shrink-revlog: improve performance: use changegroup instead of revisions
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 9712
diff changeset
    85
    unlookup = lambda x: int(x, 10)
69dca8574a6a shrink-revlog: improve performance: use changegroup instead of revisions
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 9712
diff changeset
    86
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    87
    try:
10009
69dca8574a6a shrink-revlog: improve performance: use changegroup instead of revisions
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 9712
diff changeset
    88
        group = util.chunkbuffer(r1.group(order, lookup, progress))
69dca8574a6a shrink-revlog: improve performance: use changegroup instead of revisions
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 9712
diff changeset
    89
        chunkiter = changegroup.chunkiter(group)
69dca8574a6a shrink-revlog: improve performance: use changegroup instead of revisions
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 9712
diff changeset
    90
        r2.addgroup(chunkiter, unlookup, tr)
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    91
    finally:
10496
45734b51c99b progress: mark strings for translation
Martin Geisler <mg@lazybytes.net>
parents: 10440
diff changeset
    92
        ui.progress(_('writing'), None, len(order))
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
    93
10542
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
    94
def report(ui, r1, r2):
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
    95
    def getsize(r):
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
    96
        s = 0
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
    97
        for fn in (r.indexfile, r.datafile):
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
    98
            try:
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
    99
                s += os.stat(fn).st_size
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   100
            except OSError, inst:
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   101
                if inst.errno != errno.ENOENT:
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   102
                    raise
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   103
        return s
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   104
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   105
    oldsize = float(getsize(r1))
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   106
    newsize = float(getsize(r2))
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   107
9712
18b134ef294c kill trailing whitespace
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 9515
diff changeset
   108
    # argh: have to pass an int to %d, because a float >= 2^32
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   109
    # blows up under Python 2.5 or earlier
10508
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   110
    ui.write(_('old file size: %12d bytes (%6.1f MiB)\n')
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10241
diff changeset
   111
             % (int(oldsize), oldsize / 1024 / 1024))
10508
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   112
    ui.write(_('new file size: %12d bytes (%6.1f MiB)\n')
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10241
diff changeset
   113
             % (int(newsize), newsize / 1024 / 1024))
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   114
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   115
    shrink_percent = (oldsize - newsize) / oldsize * 100
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   116
    shrink_factor = oldsize / newsize
10508
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   117
    ui.write(_('shrinkage: %.1f%% (%.1fx)\n')
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   118
             % (shrink_percent, shrink_factor))
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   119
10215
9d79b8f58bea contrib: turn shrink-revlog.py into an extension
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10214
diff changeset
   120
def shrink(ui, repo, **opts):
9d79b8f58bea contrib: turn shrink-revlog.py into an extension
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10214
diff changeset
   121
    """
9d79b8f58bea contrib: turn shrink-revlog.py into an extension
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10214
diff changeset
   122
    Shrink revlog by re-ordering revisions. Will operate on manifest for
9d79b8f58bea contrib: turn shrink-revlog.py into an extension
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10214
diff changeset
   123
    the given repository if no other revlog is specified."""
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   124
10215
9d79b8f58bea contrib: turn shrink-revlog.py into an extension
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10214
diff changeset
   125
    if not repo.local():
10508
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   126
        raise util.Abort(_('not a local repository: %s') % repo.root)
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   127
10215
9d79b8f58bea contrib: turn shrink-revlog.py into an extension
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10214
diff changeset
   128
    fn = opts.get('revlog')
9d79b8f58bea contrib: turn shrink-revlog.py into an extension
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10214
diff changeset
   129
    if not fn:
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   130
        indexfn = repo.sjoin('00manifest.i')
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   131
    else:
10215
9d79b8f58bea contrib: turn shrink-revlog.py into an extension
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10214
diff changeset
   132
        if not fn.endswith('.i'):
10508
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   133
            raise util.Abort(_('--revlog option must specify the revlog index '
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   134
                               'file (*.i), not %s') % opts.get('revlog'))
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   135
10215
9d79b8f58bea contrib: turn shrink-revlog.py into an extension
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10214
diff changeset
   136
        indexfn = os.path.realpath(fn)
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   137
        store = repo.sjoin('')
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   138
        if not indexfn.startswith(store):
10508
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   139
            raise util.Abort(_('--revlog option must specify a revlog in %s, '
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   140
                               'not %s') % (store, indexfn))
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   141
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   142
    if not os.path.exists(indexfn):
10508
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   143
        raise util.Abort(_('no such file: %s') % indexfn)
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   144
    if '00changelog' in indexfn:
10508
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   145
        raise util.Abort(_('shrinking the changelog '
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   146
                           'will corrupt your repository'))
10542
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   147
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   148
    ui.write(_('shrinking %s\n') % indexfn)
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   149
    prefix = os.path.basename(indexfn)[:-1]
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   150
    (tmpfd, tmpindexfn) = tempfile.mkstemp(dir=os.path.dirname(indexfn),
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   151
                                           prefix=prefix,
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   152
                                           suffix='.i')
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   153
    os.close(tmpfd)
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   154
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   155
    r1 = revlog.revlog(util.opener(os.getcwd(), audit=False), indexfn)
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   156
    r2 = revlog.revlog(util.opener(os.getcwd(), audit=False), tmpindexfn)
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   157
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   158
    datafn, tmpdatafn = r1.datafile, r2.datafile
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   159
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   160
    oldindexfn = indexfn + '.old'
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   161
    olddatafn = datafn + '.old'
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   162
    if os.path.exists(oldindexfn) or os.path.exists(olddatafn):
10508
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   163
        raise util.Abort(_('one or both of\n'
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   164
                           '  %s\n'
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   165
                           '  %s\n'
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   166
                           'exists from a previous run; please clean up '
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   167
                           'before running again') % (oldindexfn, olddatafn))
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   168
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   169
    # Don't use repo.transaction(), because then things get hairy with
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   170
    # paths: some need to be relative to .hg, and some need to be
10216
843f6ee6d14b contrib: small documentation fixes in shrink-revlog.py
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10215
diff changeset
   171
    # absolute. Doing it this way keeps things simple: everything is an
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   172
    # absolute path.
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   173
    lock = repo.lock(wait=False)
10234
c8d6f339bbd7 shrink-revlog: make it work on windows (issue1976)
Patrick Mezard <pmezard@gmail.com>
parents: 10230
diff changeset
   174
    tr = transaction.transaction(ui.warn,
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   175
                                 open,
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   176
                                 repo.sjoin('journal'))
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   177
10542
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   178
    def ignoremissing(func):
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   179
        def f(*args, **kw):
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   180
            try:
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   181
                return func(*args, **kw)
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   182
            except OSError, inst:
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   183
                if inst.errno != errno.ENOENT:
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   184
                    raise
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   185
        return f
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   186
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   187
    try:
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   188
        try:
10213
9e6848f352b0 contrib: use ui to write in shrink-revlog.py
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10009
diff changeset
   189
            order = toposort(ui, r1)
9e6848f352b0 contrib: use ui to write in shrink-revlog.py
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10009
diff changeset
   190
            writerevs(ui, r1, r2, order, tr)
10542
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   191
            report(ui, r1, r2)
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   192
            tr.close()
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   193
        except:
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   194
            # Abort transaction first, so we truncate the files before
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   195
            # deleting them.
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   196
            tr.abort()
10542
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   197
            for fn in (tmpindexfn, tmpdatafn):
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   198
                ignoremissing(os.unlink)(fn)
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   199
            raise
10241
4b2a086bee31 shrink-revlog: add --dry-run option
Patrick Mezard <pmezard@gmail.com>
parents: 10236
diff changeset
   200
        if not opts.get('dry_run'):
10542
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   201
            # racy, both files cannot be renamed atomically
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   202
            # copy files
10241
4b2a086bee31 shrink-revlog: add --dry-run option
Patrick Mezard <pmezard@gmail.com>
parents: 10236
diff changeset
   203
            util.os_link(indexfn, oldindexfn)
10542
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   204
            ignoremissing(util.os_link)(datafn, olddatafn)
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   205
            # rename
10241
4b2a086bee31 shrink-revlog: add --dry-run option
Patrick Mezard <pmezard@gmail.com>
parents: 10236
diff changeset
   206
            util.rename(tmpindexfn, indexfn)
10542
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   207
            try:
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   208
                util.rename(tmpdatafn, datafn)
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   209
            except OSError, inst:
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   210
                if inst.errno != errno.ENOENT:
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   211
                    raise
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   212
                ignoremissing(os.unlink)(datafn)
10241
4b2a086bee31 shrink-revlog: add --dry-run option
Patrick Mezard <pmezard@gmail.com>
parents: 10236
diff changeset
   213
        else:
10542
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   214
            for fn in (tmpindexfn, tmpdatafn):
989b2a5eaaba shrink: handle all combinations of inline/non-inline revlogs
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10509
diff changeset
   215
                ignoremissing(os.unlink)(fn)
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   216
    finally:
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   217
        lock.release()
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   218
10241
4b2a086bee31 shrink-revlog: add --dry-run option
Patrick Mezard <pmezard@gmail.com>
parents: 10236
diff changeset
   219
    if not opts.get('dry_run'):
10508
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   220
        ui.write(_('note: old revlog saved in:\n'
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   221
                   '  %s\n'
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   222
                   '  %s\n'
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   223
                   '(You can delete those files when you are satisfied that your\n'
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   224
                   'repository is still sane.  '
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   225
                   'Running \'hg verify\' is strongly recommended.)\n')
10241
4b2a086bee31 shrink-revlog: add --dry-run option
Patrick Mezard <pmezard@gmail.com>
parents: 10236
diff changeset
   226
                 % (oldindexfn, olddatafn))
9515
f7d85980261c Add script to rewrite revlog to workaround lack of parent deltas.
Greg Ward <greg-hg@gerg.ca>
parents:
diff changeset
   227
10215
9d79b8f58bea contrib: turn shrink-revlog.py into an extension
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10214
diff changeset
   228
cmdtable = {
9d79b8f58bea contrib: turn shrink-revlog.py into an extension
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10214
diff changeset
   229
    'shrink': (shrink,
10508
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   230
               [('', 'revlog', '', _('index (.i) file of the revlog to shrink')),
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   231
                ('n', 'dry-run', None, _('do not shrink, simulate only')),
10241
4b2a086bee31 shrink-revlog: add --dry-run option
Patrick Mezard <pmezard@gmail.com>
parents: 10236
diff changeset
   232
                ],
10508
cc35ad583e66 shrink-revlog: add strings for translation / import _ before using it
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 10496
diff changeset
   233
               _('hg shrink [--revlog PATH]'))
10215
9d79b8f58bea contrib: turn shrink-revlog.py into an extension
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 10214
diff changeset
   234
}
10236
49a8625b8cac shrink-revlog: help/doc tweaks
Greg Ward <greg-hg@gerg.ca>
parents: 10234
diff changeset
   235
49a8625b8cac shrink-revlog: help/doc tweaks
Greg Ward <greg-hg@gerg.ca>
parents: 10234
diff changeset
   236
if __name__ == "__main__":
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10241
diff changeset
   237
    print "shrink-revlog.py is now an extension (see hg help extensions)"