tests/test-storage.py
author Gregory Szorc <gregory.szorc@gmail.com>
Tue, 02 Oct 2018 17:34:34 -0700
changeset 40056 324b4b10351e
parent 40051 cdf61ab1f54c
child 40320 9b2e1b00ee94
permissions -rw-r--r--
revlog: rewrite censoring logic I was able to corrupt a revlog relatively easily with the existing censoring code. The underlying problem is that the existing code doesn't fully take delta chains into account. When copying revisions that occur after the censored revision, the delta base can refer to a censored revision. Then at read time, things blow up due to the revision data not being a compressed delta. This commit rewrites the revlog censoring code to take a higher-level approach. We now create a new revlog instance pointing at temp files. We iterate through each revision in the source revlog and insert those revisions into the new revlog, replacing the censored revision's data along the way. The new implementation isn't as efficient as the old one. This is because it will fully engage delta computation on insertion. But I don't think it matters. The new implementation is a bit hacky because it attempts to reload the revlog instance with a new revlog index/data file. This is fragile. But this is needed because the index (which could be backed by C) would have a cached copy of the old, possibly changed data and that could lead to problems accessing index or revision data later. One benefit of the new approach is that we integrate with the transaction. The old revlog is backed up and if the transaction is rolled back, the original revlog is restored. As part of this, we had to teach the transaction about the store vfs. I'm not super keen about this. But this was the easiest way to hook things up to the transaction. We /could/ just ignore the transaction like we were doing before. But any file mutation should be governed by transaction semantics, including undo during rollback. Differential Revision: https://phab.mercurial-scm.org/D4869
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
39772
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     1
# This test verifies the conformance of various classes to various
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     2
# storage interfaces.
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     3
from __future__ import absolute_import
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     4
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     5
import silenttestrunner
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     6
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     7
from mercurial import (
40051
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
     8
    error,
39772
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     9
    filelog,
40051
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    10
    revlog,
39772
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    11
    transaction,
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    12
    ui as uimod,
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    13
    vfs as vfsmod,
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    14
)
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    15
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    16
from mercurial.testing import (
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    17
    storage as storagetesting,
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    18
)
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    19
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    20
STATE = {
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    21
    'lastindex': 0,
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    22
    'ui': uimod.ui(),
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    23
    'vfs': vfsmod.vfs(b'.', realpath=True),
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    24
}
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    25
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    26
def makefilefn(self):
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    27
    """Factory for filelog instances."""
39953
a3a9b93bff80 py3: byteify test-storage.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39772
diff changeset
    28
    fl = filelog.filelog(STATE['vfs'], b'filelog-%d' % STATE['lastindex'])
39772
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    29
    STATE['lastindex'] += 1
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    30
    return fl
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    31
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    32
def maketransaction(self):
40056
324b4b10351e revlog: rewrite censoring logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40051
diff changeset
    33
    vfsmap = {'plain': STATE['vfs'], 'store': STATE['vfs']}
39772
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    34
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    35
    return transaction.transaction(STATE['ui'].warn, STATE['vfs'], vfsmap,
39953
a3a9b93bff80 py3: byteify test-storage.py
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39772
diff changeset
    36
                                   b'journal', b'undo')
39772
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    37
40051
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    38
def addrawrevision(self, fl, tr, node, p1, p2, linkrev, rawtext=None,
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    39
                   delta=None, censored=False, ellipsis=False, extstored=False):
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    40
    flags = 0
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    41
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    42
    if censored:
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    43
        flags |= revlog.REVIDX_ISCENSORED
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    44
    if ellipsis:
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    45
        flags |= revlog.REVIDX_ELLIPSIS
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    46
    if extstored:
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    47
        flags |= revlog.REVIDX_EXTSTORED
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    48
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    49
    if rawtext is not None:
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    50
        fl._revlog.addrawrevision(rawtext, tr, linkrev, p1, p2, node, flags)
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    51
    elif delta is not None:
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    52
        raise error.Abort('support for storing raw deltas not yet supported')
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    53
    else:
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    54
        raise error.Abort('must supply rawtext or delta arguments')
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    55
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    56
    # We may insert bad data. Clear caches to prevent e.g. cache hits to
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    57
    # bypass hash verification.
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    58
    fl._revlog.clearcaches()
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    59
39772
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    60
# Assigning module-level attributes that inherit from unittest.TestCase
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    61
# is all that is needed to register tests.
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    62
filelogindextests = storagetesting.makeifileindextests(makefilefn,
40051
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    63
                                                       maketransaction,
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    64
                                                       addrawrevision)
39772
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    65
filelogdatatests = storagetesting.makeifiledatatests(makefilefn,
40051
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    66
                                                     maketransaction,
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    67
                                                     addrawrevision)
39772
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    68
filelogmutationtests = storagetesting.makeifilemutationtests(makefilefn,
40051
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    69
                                                             maketransaction,
cdf61ab1f54c testing: add file storage integration for bad hashes and censoring
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39953
diff changeset
    70
                                                             addrawrevision)
39772
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    71
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    72
if __name__ == '__main__':
ae531f5e583c testing: add interface unit tests for file storage
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    73
    silenttestrunner.main(__name__)