mercurial/revlogutils/nodemap.py
author Pierre-Yves David <pierre-yves.david@octobus.net>
Thu, 07 Mar 2024 04:15:23 +0100
changeset 51536 718f28ea3af4
parent 51265 85d96517e650
permissions -rw-r--r--
branchcache: add a "pure topological head" fast path In a narrow but actually quick common case, all topological heads are all on the same branch and all open. In this case, computing the branch map is very simple. We can quickly detect situation where this situation will not change. So we update the V3 format to be able to express this situation and upgrade the update code to detect we remains in that mode. The branch cache is populated with the actual value when the branch map is accessed, but the update_disk method can do the update without needing to populate it.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
44034
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     1
# nodemap.py - nodemap related code and utilities
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     2
#
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     3
# Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     4
# Copyright 2019 George Racinet <georges.racinet@octobus.net>
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     5
#
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     6
# This software may be used and distributed according to the terms of the
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     7
# GNU General Public License version 2 or any later version.
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
     8
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
     9
44312
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
    10
import re
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
    11
import struct
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
    12
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46089
diff changeset
    13
from ..node import hex
44788
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44745
diff changeset
    14
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
    15
from .. import (
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
    16
    error,
48693
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    17
    requirements,
44314
7f4f7ef3133e nodemap: add a optional `nodemap_add_full` method on indexes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44312
diff changeset
    18
    util,
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
    19
)
47312
7ea39d633cf3 docket: move the uid logic in the `revlogutils.docket` module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47311
diff changeset
    20
from . import docket as docket_mod
44034
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    21
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    22
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    23
class NodeMap(dict):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    24
    def __missing__(self, x):
ab595920de0e revlogutils: move the NodeMap class in a dedicated nodemap module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents:
diff changeset
    25
        raise error.RevlogError(b'unknown node: %s' % x)
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
    26
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
    27
48038
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    28
def test_race_hook_1():
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    29
    """hook point for test
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    30
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    31
    This let tests to have things happens between the docket reading and the
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    32
    data reading"""
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    33
    pass
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    34
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    35
48693
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    36
def post_stream_cleanup(repo):
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    37
    """The stream clone might needs to remove some file if persisten nodemap
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    38
    was dropped while stream cloning
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    39
    """
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    40
    if requirements.REVLOGV1_REQUIREMENT not in repo.requirements:
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    41
        return
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    42
    if requirements.NODEMAP_REQUIREMENT in repo.requirements:
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    43
        return
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    44
    unfi = repo.unfiltered()
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    45
    delete_nodemap(None, unfi, unfi.changelog)
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    46
    delete_nodemap(None, repo, unfi.manifestlog._rootstore._revlog)
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    47
de3ac3d2c60b stream-clone: allow to change persistent-nodemap format during stream clone
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48690
diff changeset
    48
44309
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44308
diff changeset
    49
def persisted_data(revlog):
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44308
diff changeset
    50
    """read the nodemap for a revlog from disk"""
47164
04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47152
diff changeset
    51
    if revlog._nodemap_file is None:
44309
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44308
diff changeset
    52
        return None
47164
04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47152
diff changeset
    53
    pdata = revlog.opener.tryread(revlog._nodemap_file)
44311
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
    54
    if not pdata:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
    55
        return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
    56
    offset = 0
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
    57
    (version,) = S_VERSION.unpack(pdata[offset : offset + S_VERSION.size])
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
    58
    if version != ONDISK_VERSION:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
    59
        return None
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
    60
    offset += S_VERSION.size
44335
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44333
diff changeset
    61
    headers = S_HEADER.unpack(pdata[offset : offset + S_HEADER.size])
44515
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
    62
    uid_size, tip_rev, data_length, data_unused, tip_node_size = headers
44311
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
    63
    offset += S_HEADER.size
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
    64
    docket = NodeMapDocket(pdata[offset : offset + uid_size])
44515
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
    65
    offset += uid_size
44335
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44333
diff changeset
    66
    docket.tip_rev = tip_rev
44515
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
    67
    docket.tip_node = pdata[offset : offset + tip_node_size]
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
    68
    docket.data_length = data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
    69
    docket.data_unused = data_unused
44311
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
    70
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
    71
    filename = _rawdata_filepath(revlog, docket)
44792
5e3c718692bb nodemap: drop the 'exp-' prefix for internal opener option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44790
diff changeset
    72
    use_mmap = revlog.opener.options.get(b"persistent-nodemap.mmap")
48038
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    73
52018f8ef020 persistent-nodemap: introduce a test to highlight possible race
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47451
diff changeset
    74
    test_race_hook_1()
44363
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
    75
    try:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
    76
        with revlog.opener(filename) as fd:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
    77
            if use_mmap:
46971
a3720569a43f nodemap: deal with data mmap error
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46794
diff changeset
    78
                try:
a3720569a43f nodemap: deal with data mmap error
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46794
diff changeset
    79
                    data = util.buffer(util.mmapread(fd, data_length))
a3720569a43f nodemap: deal with data mmap error
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46794
diff changeset
    80
                except ValueError:
a3720569a43f nodemap: deal with data mmap error
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46794
diff changeset
    81
                    # raised when the read file is too small
a3720569a43f nodemap: deal with data mmap error
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46794
diff changeset
    82
                    data = b''
44363
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
    83
            else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
    84
                data = fd.read(data_length)
49306
2e726c934fcd py3: catch FileNotFoundError instead of checking errno == ENOENT
Manuel Jacob <me@manueljacob.de>
parents: 48946
diff changeset
    85
    except FileNotFoundError:
2e726c934fcd py3: catch FileNotFoundError instead of checking errno == ENOENT
Manuel Jacob <me@manueljacob.de>
parents: 48946
diff changeset
    86
        return None
44339
c7eebdb15139 nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44338
diff changeset
    87
    if len(data) < data_length:
c7eebdb15139 nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44338
diff changeset
    88
        return None
c7eebdb15139 nodemap: never read more than the expected data amount
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44338
diff changeset
    89
    return docket, data
44309
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44308
diff changeset
    90
6c07480d6659 nodemap: add a function to read the data from disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44308
diff changeset
    91
44308
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
    92
def setup_persistent_nodemap(tr, revlog):
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
    93
    """Install whatever is needed transaction side to persist a nodemap on disk
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
    94
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
    95
    (only actually persist the nodemap if this is relevant for this revlog)
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
    96
    """
44310
daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44309
diff changeset
    97
    if revlog._inline:
daad3aace942 nodemap: only use persistent nodemap for non-inlined revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44309
diff changeset
    98
        return  # inlined revlog are too small for this to be relevant
47164
04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47152
diff changeset
    99
    if revlog._nodemap_file is None:
44308
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   100
        return  # we do not use persistent_nodemap on this revlog
44517
448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44516
diff changeset
   101
448d700e0d27 nodemap: make sure the nodemap docket is updated after the changelog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44516
diff changeset
   102
    # we need to happen after the changelog finalization, in that use "cl-"
47164
04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47152
diff changeset
   103
    callback_id = b"nm-revlog-persistent-nodemap-%s" % revlog._nodemap_file
44308
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   104
    if tr.hasfinalize(callback_id):
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   105
        return  # no need to register again
44516
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44515
diff changeset
   106
    tr.addpending(
46468
83f037acf71a nodemap: make `_persist_nodemap` a public function
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46321
diff changeset
   107
        callback_id, lambda tr: persist_nodemap(tr, revlog, pending=True)
44516
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44515
diff changeset
   108
    )
46468
83f037acf71a nodemap: make `_persist_nodemap` a public function
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46321
diff changeset
   109
    tr.addfinalize(callback_id, lambda tr: persist_nodemap(tr, revlog))
44497
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   110
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   111
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48875
diff changeset
   112
class _NoTransaction:
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44792
diff changeset
   113
    """transaction like object to update the nodemap outside a transaction"""
44497
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   114
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   115
    def __init__(self):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   116
        self._postclose = {}
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   117
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   118
    def addpostclose(self, callback_id, callback_func):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   119
        self._postclose[callback_id] = callback_func
44308
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   120
44634
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   121
    def registertmp(self, *args, **kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   122
        pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   123
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   124
    def addbackup(self, *args, **kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   125
        pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   126
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   127
    def add(self, *args, **kwargs):
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   128
        pass
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   129
44635
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   130
    def addabort(self, *args, **kwargs):
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   131
        pass
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   132
44788
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44745
diff changeset
   133
    def _report(self, *args):
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44745
diff changeset
   134
        pass
6493f0a567c2 nodemap: add a new mode option, with an optional "warn" value
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44745
diff changeset
   135
44308
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   136
44445
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44363
diff changeset
   137
def update_persistent_nodemap(revlog):
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44363
diff changeset
   138
    """update the persistent nodemap right now
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44363
diff changeset
   139
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44363
diff changeset
   140
    To be used for updating the nodemap on disk outside of a normal transaction
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44363
diff changeset
   141
    setup (eg, `debugupdatecache`).
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44363
diff changeset
   142
    """
44745
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
   143
    if revlog._inline:
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
   144
        return  # inlined revlog are too small for this to be relevant
47164
04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47152
diff changeset
   145
    if revlog._nodemap_file is None:
44745
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
   146
        return  # we do not use persistent_nodemap on this revlog
b4537125eb3c nodemap: skip persistent nodemap warming for revlog not using it
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44697
diff changeset
   147
44497
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   148
    notr = _NoTransaction()
46468
83f037acf71a nodemap: make `_persist_nodemap` a public function
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46321
diff changeset
   149
    persist_nodemap(notr, revlog)
44497
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   150
    for k in sorted(notr._postclose):
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   151
        notr._postclose[k](None)
44445
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44363
diff changeset
   152
336ec75ed1ac nodemap: warm the persistent nodemap on disk with debugupdatecache
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44363
diff changeset
   153
46526
67b5fafd3a46 upgrade: speed up when we have only nodemap to downgrade
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46472
diff changeset
   154
def delete_nodemap(tr, repo, revlog):
47451
f70ca39d0ab8 nodemap: fix some comment formatting
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47312
diff changeset
   155
    """Delete nodemap data on disk for a given revlog"""
48690
d55c4472bbb6 persistent-nodemap: properly delete all nodemap files on downgrade
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48424
diff changeset
   156
    prefix = revlog.radix
d55c4472bbb6 persistent-nodemap: properly delete all nodemap files on downgrade
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48424
diff changeset
   157
    pattern = re.compile(br"(^|/)%s(-[0-9a-f]+\.nd|\.n(\.a)?)$" % prefix)
d55c4472bbb6 persistent-nodemap: properly delete all nodemap files on downgrade
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48424
diff changeset
   158
    dirpath = revlog.opener.dirname(revlog._indexfile)
d55c4472bbb6 persistent-nodemap: properly delete all nodemap files on downgrade
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48424
diff changeset
   159
    for f in revlog.opener.listdir(dirpath):
d55c4472bbb6 persistent-nodemap: properly delete all nodemap files on downgrade
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48424
diff changeset
   160
        if pattern.match(f):
d55c4472bbb6 persistent-nodemap: properly delete all nodemap files on downgrade
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 48424
diff changeset
   161
            repo.svfs.tryunlink(f)
46526
67b5fafd3a46 upgrade: speed up when we have only nodemap to downgrade
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46472
diff changeset
   162
67b5fafd3a46 upgrade: speed up when we have only nodemap to downgrade
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46472
diff changeset
   163
46472
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46470
diff changeset
   164
def persist_nodemap(tr, revlog, pending=False, force=False):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44792
diff changeset
   165
    """Write nodemap data on disk for a given revlog"""
51175
1486d8c63f64 persistent-nodemap: avoid writing nodemap for empty revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50928
diff changeset
   166
    if len(revlog.index) <= 0:
1486d8c63f64 persistent-nodemap: avoid writing nodemap for empty revlog
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50928
diff changeset
   167
        return
44308
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   168
    if getattr(revlog, 'filteredrevs', ()):
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   169
        raise error.ProgrammingError(
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   170
            "cannot persist nodemap of a filtered changelog"
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   171
        )
47164
04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47152
diff changeset
   172
    if revlog._nodemap_file is None:
46472
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46470
diff changeset
   173
        if force:
47164
04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47152
diff changeset
   174
            revlog._nodemap_file = get_nodemap_file(revlog)
46472
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46470
diff changeset
   175
        else:
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46470
diff changeset
   176
            msg = "calling persist nodemap on a revlog without the feature enabled"
98e39f04d60e upgrade: implement partial upgrade for upgrading persistent-nodemap
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46470
diff changeset
   177
            raise error.ProgrammingError(msg)
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   178
50928
d718eddf01d9 safehasattr: drop usage in favor of hasattr
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50510
diff changeset
   179
    can_incremental = hasattr(revlog.index, "nodemap_data_incremental")
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   180
    ondisk_docket = revlog._nodemap_docket
50928
d718eddf01d9 safehasattr: drop usage in favor of hasattr
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50510
diff changeset
   181
    feed_data = hasattr(revlog.index, "update_nodemap_data")
44792
5e3c718692bb nodemap: drop the 'exp-' prefix for internal opener option
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44790
diff changeset
   182
    use_mmap = revlog.opener.options.get(b"persistent-nodemap.mmap")
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   183
44337
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   184
    data = None
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   185
    # first attemp an incremental update of the data
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   186
    if can_incremental and ondisk_docket is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   187
        target_docket = revlog._nodemap_docket.copy()
44337
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   188
        (
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   189
            src_docket,
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   190
            data_changed_count,
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   191
            data,
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   192
        ) = revlog.index.nodemap_data_incremental()
44636
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44635
diff changeset
   193
        new_length = target_docket.data_length + len(data)
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44635
diff changeset
   194
        new_unused = target_docket.data_unused + data_changed_count
44337
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   195
        if src_docket != target_docket:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   196
            data = None
44636
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44635
diff changeset
   197
        elif new_length <= (new_unused * 10):  # under 10% of unused data
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44635
diff changeset
   198
            data = None
44337
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   199
        else:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   200
            datafile = _rawdata_filepath(revlog, target_docket)
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   201
            # EXP-TODO: if this is a cache, this should use a cache vfs, not a
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   202
            # store vfs
44634
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   203
            tr.add(datafile, target_docket.data_length)
44338
2ea6a67ff502 nodemap: write new data from the expected current data length
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44337
diff changeset
   204
            with revlog.opener(datafile, b'r+') as fd:
2ea6a67ff502 nodemap: write new data from the expected current data length
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44337
diff changeset
   205
                fd.seek(target_docket.data_length)
44337
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   206
                fd.write(data)
44363
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   207
                if feed_data:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   208
                    if use_mmap:
51265
85d96517e650 persistent-nodemap: respect the mmap setting when refreshing data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51175
diff changeset
   209
                        fd.flush()
85d96517e650 persistent-nodemap: respect the mmap setting when refreshing data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51175
diff changeset
   210
                        new_data = util.buffer(util.mmapread(fd, new_length))
85d96517e650 persistent-nodemap: respect the mmap setting when refreshing data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51175
diff changeset
   211
                    else:
44363
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   212
                        fd.seek(0)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   213
                        new_data = fd.read(new_length)
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   214
            target_docket.data_length = new_length
44636
c70bcaf7927b nodemap: automatically "vacuum" the persistent nodemap when too sparse
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44635
diff changeset
   215
            target_docket.data_unused = new_unused
44337
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   216
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   217
    if data is None:
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   218
        # otherwise fallback to a full new export
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   219
        target_docket = NodeMapDocket()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   220
        datafile = _rawdata_filepath(revlog, target_docket)
50928
d718eddf01d9 safehasattr: drop usage in favor of hasattr
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50510
diff changeset
   221
        if hasattr(revlog.index, "nodemap_data_all"):
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   222
            data = revlog.index.nodemap_data_all()
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   223
        else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   224
            data = persistent_data(revlog.index)
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   225
        # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   226
        # store vfs
44635
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   227
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   228
        tryunlink = revlog.opener.tryunlink
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   229
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   230
        def abortck(tr):
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   231
            tryunlink(datafile)
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   232
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   233
        callback_id = b"delete-%s" % datafile
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   234
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   235
        # some flavor of the transaction abort does not cleanup new file, it
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   236
        # simply empty them.
99ea74cbed74 nodemap: display percentage of unused in `hg debugnodemap`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44634
diff changeset
   237
        tr.addabort(callback_id, abortck)
44363
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   238
        with revlog.opener(datafile, b'w+') as fd:
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   239
            fd.write(data)
44363
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   240
            if feed_data:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   241
                if use_mmap:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   242
                    new_data = data
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   243
                else:
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   244
                    fd.flush()
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   245
                    new_data = util.buffer(util.mmapread(fd, len(data)))
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   246
        target_docket.data_length = len(data)
44335
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44333
diff changeset
   247
    target_docket.tip_rev = revlog.tiprev()
44515
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   248
    target_docket.tip_node = revlog.node(target_docket.tip_rev)
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   249
    # EXP-TODO: if this is a cache, this should use a cache vfs, not a
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   250
    # store vfs
47164
04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47152
diff changeset
   251
    file_path = revlog._nodemap_file
44516
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44515
diff changeset
   252
    if pending:
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44515
diff changeset
   253
        file_path += b'.a'
44634
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   254
        tr.registertmp(file_path)
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   255
    else:
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   256
        tr.addbackup(file_path)
01b0805534bb nodemap: make sure on disk change get rolled back with the transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44517
diff changeset
   257
44516
64e2f603de9d nodemap: make sure hooks have access to an up-to-date version
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44515
diff changeset
   258
    with revlog.opener(file_path, b'w', atomictemp=True) as fp:
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   259
        fp.write(target_docket.serialize())
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   260
    revlog._nodemap_docket = target_docket
44363
f7459da77f23 nodemap: introduce an option to use mmap to read the nodemap mapping
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44340
diff changeset
   261
    if feed_data:
44340
6ecc34b31137 nodemap: update the index with the newly written data (when appropriate)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44339
diff changeset
   262
        revlog.index.update_nodemap_data(target_docket, new_data)
6ecc34b31137 nodemap: update the index with the newly written data (when appropriate)
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44339
diff changeset
   263
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   264
    # search for old index file in all cases, some older process might have
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   265
    # left one behind.
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   266
    olds = _other_rawdata_filepath(revlog, target_docket)
44312
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   267
    if olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   268
        realvfs = getattr(revlog, '_realopener', revlog.opener)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   269
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   270
        def cleanup(tr):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   271
            for oldfile in olds:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   272
                realvfs.tryunlink(oldfile)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   273
47164
04f2f94836a5 revlog: rename `nodemap_file` to `_nodemap_file`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47152
diff changeset
   274
        callback_id = b"revlog-cleanup-nodemap-%s" % revlog._nodemap_file
44497
3265c92f7d13 nodemap: deal with the "debugupdatecache" case using a "fake" transaction
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44494
diff changeset
   275
        tr.addpostclose(callback_id, cleanup)
44311
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   276
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   277
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   278
### Nodemap docket file
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   279
#
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   280
# The nodemap data are stored on disk using 2 files:
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   281
#
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   282
# * a raw data files containing a persistent nodemap
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   283
#   (see `Nodemap Trie` section)
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   284
#
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   285
# * a small "docket" file containing medatadata
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   286
#
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   287
# While the nodemap data can be multiple tens of megabytes, the "docket" is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   288
# small, it is easy to update it automatically or to duplicated its content
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   289
# during a transaction.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   290
#
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   291
# Multiple raw data can exist at the same time (The currently valid one and a
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   292
# new one beind used by an in progress transaction). To accomodate this, the
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   293
# filename hosting the raw data has a variable parts. The exact filename is
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   294
# specified inside the "docket" file.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   295
#
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   296
# The docket file contains information to find, qualify and validate the raw
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   297
# data. Its content is currently very light, but it will expand as the on disk
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   298
# nodemap gains the necessary features to be used in production.
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   299
44790
261e71752d1f nodemap: move on disk file to version 1
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44789
diff changeset
   300
ONDISK_VERSION = 1
44311
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   301
S_VERSION = struct.Struct(">B")
44515
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   302
S_HEADER = struct.Struct(">BQQQQ")
44311
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   303
47310
9a3aa54774ff persistent-nodemap: add a way to make the picked uid predictable
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47240
diff changeset
   304
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48875
diff changeset
   305
class NodeMapDocket:
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   306
    """metadata associated with persistent nodemap data
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   307
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   308
    The persistent data may come from disk or be on their way to disk.
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   309
    """
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   310
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   311
    def __init__(self, uid=None):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   312
        if uid is None:
47312
7ea39d633cf3 docket: move the uid logic in the `revlogutils.docket` module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47311
diff changeset
   313
            uid = docket_mod.make_uid()
44494
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   314
        # a unique identifier for the data file:
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   315
        #   - When new data are appended, it is preserved.
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   316
        #   - When a new data file is created, a new identifier is generated.
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   317
        self.uid = uid
44494
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   318
        # the tipmost revision stored in the data file. This revision and all
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   319
        # revision before it are expected to be encoded in the data file.
44335
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44333
diff changeset
   320
        self.tip_rev = None
44515
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   321
        # the node of that tipmost revision, if it mismatch the current index
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   322
        # data the docket is not valid for the current index and should be
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   323
        # discarded.
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   324
        #
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   325
        # note: this method is not perfect as some destructive operation could
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   326
        # preserve the same tip_rev + tip_node while altering lower revision.
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   327
        # However this multiple other caches have the same vulnerability (eg:
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   328
        # brancmap cache).
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   329
        self.tip_node = None
44494
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   330
        # the size (in bytes) of the persisted data to encode the nodemap valid
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   331
        # for `tip_rev`.
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   332
        #   - data file shorter than this are corrupted,
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   333
        #   - any extra data should be ignored.
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   334
        self.data_length = None
44494
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   335
        # the amount (in bytes) of "dead" data, still in the data file but no
283fd803afa5 nodemap: document the docket attributes
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44462
diff changeset
   336
        # longer used for the nodemap.
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   337
        self.data_unused = 0
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   338
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   339
    def copy(self):
44335
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44333
diff changeset
   340
        new = NodeMapDocket(uid=self.uid)
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44333
diff changeset
   341
        new.tip_rev = self.tip_rev
44515
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   342
        new.tip_node = self.tip_node
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   343
        new.data_length = self.data_length
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   344
        new.data_unused = self.data_unused
44335
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44333
diff changeset
   345
        return new
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   346
44337
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   347
    def __cmp__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   348
        if self.uid < other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   349
            return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   350
        if self.uid > other.uid:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   351
            return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   352
        elif self.data_length < other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   353
            return -1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   354
        elif self.data_length > other.data_length:
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   355
            return 1
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   356
        return 0
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   357
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   358
    def __eq__(self, other):
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   359
        return self.uid == other.uid and self.data_length == other.data_length
1d2b37def017 nodemap: double check the source docket when doing incremental update
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44336
diff changeset
   360
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   361
    def serialize(self):
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   362
        """return serialized bytes for a docket using the passed uid"""
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   363
        data = []
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   364
        data.append(S_VERSION.pack(ONDISK_VERSION))
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   365
        headers = (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   366
            len(self.uid),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   367
            self.tip_rev,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   368
            self.data_length,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   369
            self.data_unused,
44515
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   370
            len(self.tip_node),
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   371
        )
44335
e41a164db7a9 nodemap: track the maximum revision tracked in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44333
diff changeset
   372
        data.append(S_HEADER.pack(*headers))
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   373
        data.append(self.uid)
44515
6c906eaedd0d nodemap: track the tip_node for validation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44497
diff changeset
   374
        data.append(self.tip_node)
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   375
        return b''.join(data)
44311
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   376
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   377
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   378
def _rawdata_filepath(revlog, docket):
44311
2b72c4ff8ed1 nodemap: use an intermediate "docket" file to carry small metadata
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44310
diff changeset
   379
    """The (vfs relative) nodemap's rawdata file for a given uid"""
47152
09338a2d5c14 revlog: also use radix when computing nodemap data file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47151
diff changeset
   380
    prefix = revlog.radix
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   381
    return b"%s-%s.nd" % (prefix, docket.uid)
44308
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   382
5962fd0d1045 nodemap: write nodemap data on disk
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44307
diff changeset
   383
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   384
def _other_rawdata_filepath(revlog, docket):
47152
09338a2d5c14 revlog: also use radix when computing nodemap data file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47151
diff changeset
   385
    prefix = revlog.radix
44462
6aee0647e026 nodemap: fix missing r-prefix on regular expression
Augie Fackler <augie@google.com>
parents: 44445
diff changeset
   386
    pattern = re.compile(br"(^|/)%s-[0-9a-f]+\.nd$" % prefix)
44322
72c15641c8b4 nodemap: introduce an explicit class/object for the docket
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44321
diff changeset
   387
    new_file_path = _rawdata_filepath(revlog, docket)
44312
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   388
    new_file_name = revlog.opener.basename(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   389
    dirpath = revlog.opener.dirname(new_file_path)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   390
    others = []
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   391
    for f in revlog.opener.listdir(dirpath):
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   392
        if pattern.match(f) and f != new_file_name:
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   393
            others.append(f)
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   394
    return others
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   395
563dfdfd01a4 nodemap: delete older raw data file when creating a new ones
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44311
diff changeset
   396
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   397
### Nodemap Trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   398
#
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   399
# This is a simple reference implementation to compute and persist a nodemap
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   400
# trie. This reference implementation is write only. The python version of this
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   401
# is not expected to be actually used, since it wont provide performance
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   402
# improvement over existing non-persistent C implementation.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   403
#
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   404
# The nodemap is persisted as Trie using 4bits-address/16-entries block. each
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   405
# revision can be adressed using its node shortest prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   406
#
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   407
# The trie is stored as a sequence of block. Each block contains 16 entries
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   408
# (signed 64bit integer, big endian). Each entry can be one of the following:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   409
#
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   410
#  * value >=  0 -> index of sub-block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   411
#  * value == -1 -> no value
46199
8e7ce6555ea7 nodemap: match comment to actual code
Joerg Sonnenberger <joerg@bec.de>
parents: 46113
diff changeset
   412
#  * value <  -1 -> encoded revision: rev = -(value+2)
8e7ce6555ea7 nodemap: match comment to actual code
Joerg Sonnenberger <joerg@bec.de>
parents: 46113
diff changeset
   413
#
8e7ce6555ea7 nodemap: match comment to actual code
Joerg Sonnenberger <joerg@bec.de>
parents: 46113
diff changeset
   414
# See REV_OFFSET and _transform_rev below.
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   415
#
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   416
# The implementation focus on simplicity, not on performance. A Rust
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   417
# implementation should provide a efficient version of the same binary
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   418
# persistence. This reference python implementation is never meant to be
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   419
# extensively use in production.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   420
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   421
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   422
def persistent_data(index):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44792
diff changeset
   423
    """return the persistent binary form for a nodemap for a given index"""
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   424
    trie = _build_trie(index)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   425
    return _persist_trie(trie)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   426
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   427
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   428
def update_persistent_data(index, root, max_idx, last_rev):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44792
diff changeset
   429
    """return the incremental update for persistent nodemap from a given index"""
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   430
    changed_block, trie = _update_trie(index, root, last_rev)
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   431
    return (
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   432
        changed_block * S_BLOCK.size,
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   433
        _persist_trie(trie, existing_idx=max_idx),
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   434
    )
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   435
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   436
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   437
S_BLOCK = struct.Struct(">" + ("l" * 16))
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   438
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   439
NO_ENTRY = -1
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   440
# rev 0 need to be -2 because 0 is used by block, -1 is a special value.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   441
REV_OFFSET = 2
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   442
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   443
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   444
def _transform_rev(rev):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   445
    """Return the number used to represent the rev in the tree.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   446
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   447
    (or retrieve a rev number from such representation)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   448
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   449
    Note that this is an involution, a function equal to its inverse (i.e.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   450
    which gives the identity when applied to itself).
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   451
    """
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   452
    return -(rev + REV_OFFSET)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   453
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   454
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   455
def _to_int(hex_digit):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   456
    """turn an hexadecimal digit into a proper integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   457
    return int(hex_digit, 16)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   458
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   459
44315
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44314
diff changeset
   460
class Block(dict):
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44314
diff changeset
   461
    """represent a block of the Trie
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44314
diff changeset
   462
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44314
diff changeset
   463
    contains up to 16 entry indexed from 0 to 15"""
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44314
diff changeset
   464
44321
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44319
diff changeset
   465
    def __init__(self):
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44319
diff changeset
   466
        super(Block, self).__init__()
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44319
diff changeset
   467
        # If this block exist on disk, here is its ID
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44319
diff changeset
   468
        self.ondisk_id = None
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44319
diff changeset
   469
44316
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44315
diff changeset
   470
    def __iter__(self):
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44315
diff changeset
   471
        return iter(self.get(i) for i in range(16))
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44315
diff changeset
   472
44315
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44314
diff changeset
   473
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   474
def _build_trie(index):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   475
    """build a nodemap trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   476
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   477
    The nodemap stores revision number for each unique prefix.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   478
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   479
    Each block is a dictionary with keys in `[0, 15]`. Values are either
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   480
    another block or a revision number.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   481
    """
44315
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44314
diff changeset
   482
    root = Block()
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   483
    for rev in range(len(index)):
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46089
diff changeset
   484
        current_hex = hex(index[rev][7])
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46089
diff changeset
   485
        _insert_into_block(index, 0, root, rev, current_hex)
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   486
    return root
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   487
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   488
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   489
def _update_trie(index, root, last_rev):
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   490
    """consume"""
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   491
    changed = 0
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   492
    for rev in range(last_rev + 1, len(index)):
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46089
diff changeset
   493
        current_hex = hex(index[rev][7])
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46089
diff changeset
   494
        changed += _insert_into_block(index, 0, root, rev, current_hex)
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   495
    return changed, root
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   496
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   497
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   498
def _insert_into_block(index, level, block, current_rev, current_hex):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   499
    """insert a new revision in a block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   500
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   501
    index: the index we are adding revision for
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   502
    level: the depth of the current block in the trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   503
    block: the block currently being considered
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   504
    current_rev: the revision number we are adding
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   505
    current_hex: the hexadecimal representation of the of that revision
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   506
    """
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   507
    changed = 1
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   508
    if block.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   509
        block.ondisk_id = None
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   510
    hex_digit = _to_int(current_hex[level : level + 1])
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   511
    entry = block.get(hex_digit)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   512
    if entry is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   513
        # no entry, simply store the revision number
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   514
        block[hex_digit] = current_rev
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   515
    elif isinstance(entry, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   516
        # need to recurse to an underlying block
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   517
        changed += _insert_into_block(
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   518
            index, level + 1, entry, current_rev, current_hex
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   519
        )
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   520
    else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   521
        # collision with a previously unique prefix, inserting new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   522
        # vertices to fit both entry.
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46089
diff changeset
   523
        other_hex = hex(index[entry][7])
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   524
        other_rev = entry
44315
7762a295fd4d nodemap: use an explicit "Block" object in the reference implementation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44314
diff changeset
   525
        new = Block()
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   526
        block[hex_digit] = new
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   527
        _insert_into_block(index, level + 1, new, other_rev, other_hex)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   528
        _insert_into_block(index, level + 1, new, current_rev, current_hex)
44336
8374b69aef75 nodemap: track the total and unused amount of data in the rawdata file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44335
diff changeset
   529
    return changed
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   530
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   531
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   532
def _persist_trie(root, existing_idx=None):
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   533
    """turn a nodemap trie into persistent binary data
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   534
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   535
    See `_build_trie` for nodemap trie structure"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   536
    block_map = {}
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   537
    if existing_idx is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   538
        base_idx = existing_idx + 1
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   539
    else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   540
        base_idx = 0
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   541
    chunks = []
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   542
    for tn in _walk_trie(root):
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   543
        if tn.ondisk_id is not None:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   544
            block_map[id(tn)] = tn.ondisk_id
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   545
        else:
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   546
            block_map[id(tn)] = len(chunks) + base_idx
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   547
            chunks.append(_persist_block(tn, block_map))
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   548
    return b''.join(chunks)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   549
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   550
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   551
def _walk_trie(block):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   552
    """yield all the block in a trie
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   553
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   554
    Children blocks are always yield before their parent block.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   555
    """
44696
ed475420e0af nodemap: rename a variable to prevent shawoding "_"
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44636
diff changeset
   556
    for (__, item) in sorted(block.items()):
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   557
        if isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   558
            for sub_block in _walk_trie(item):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   559
                yield sub_block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   560
    yield block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   561
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   562
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   563
def _persist_block(block_node, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   564
    """produce persistent binary data for a single block
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   565
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   566
    Children block are assumed to be already persisted and present in
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   567
    block_map.
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   568
    """
44316
55b12f2593c1 nodemap: move the iteratio inside the Block object
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44315
diff changeset
   569
    data = tuple(_to_value(v, block_map) for v in block_node)
44307
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   570
    return S_BLOCK.pack(*data)
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   571
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   572
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   573
def _to_value(item, block_map):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   574
    """persist any value as an integer"""
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   575
    if item is None:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   576
        return NO_ENTRY
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   577
    elif isinstance(item, dict):
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   578
        return block_map[id(item)]
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   579
    else:
c577bb4a04d4 nodemap: have some python code writing a nodemap in persistent binary form
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44034
diff changeset
   580
        return _transform_rev(item)
44317
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   581
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   582
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   583
def parse_data(data):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   584
    """parse parse nodemap data into a nodemap Trie"""
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   585
    if (len(data) % S_BLOCK.size) != 0:
46684
98c816e89fac nodemap: convert error message to bytes
Matt Harbison <matt_harbison@yahoo.com>
parents: 46321
diff changeset
   586
        msg = b"nodemap data size is not a multiple of block size (%d): %d"
44317
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   587
        raise error.Abort(msg % (S_BLOCK.size, len(data)))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   588
    if not data:
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   589
        return Block(), None
44317
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   590
    block_map = {}
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   591
    new_blocks = []
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   592
    for i in range(0, len(data), S_BLOCK.size):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   593
        block = Block()
44321
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44319
diff changeset
   594
        block.ondisk_id = len(block_map)
f0862ee1a31e nodemap: keep track of the ondisk id of nodemap blocks
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44319
diff changeset
   595
        block_map[block.ondisk_id] = block
44317
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   596
        block_data = data[i : i + S_BLOCK.size]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   597
        values = S_BLOCK.unpack(block_data)
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   598
        new_blocks.append((block, values))
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   599
    for b, values in new_blocks:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   600
        for idx, v in enumerate(values):
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   601
            if v == NO_ENTRY:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   602
                continue
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   603
            elif v >= 0:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   604
                b[idx] = block_map[v]
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   605
            else:
78721bbdb2ab nodemap: code to parse the persistent binary nodemap data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44316
diff changeset
   606
                b[idx] = _transform_rev(v)
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   607
    return block, i // S_BLOCK.size
44318
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   608
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   609
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   610
# debug utility
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   611
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   612
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   613
def check_data(ui, index, data):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   614
    """verify that the provided nodemap data are valid for the given idex"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   615
    ret = 0
50510
4fe46f96b56b cli: fix spelling in `debugnodemap` error messages
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49306
diff changeset
   616
    ui.status((b"revisions in index:   %d\n") % len(index))
44333
50ad851efd9b nodemap: introduce append-only incremental update of the persistent data
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44323
diff changeset
   617
    root, __ = parse_data(data)
44318
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   618
    all_revs = set(_all_revisions(root))
50510
4fe46f96b56b cli: fix spelling in `debugnodemap` error messages
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49306
diff changeset
   619
    ui.status((b"revisions in nodemap: %d\n") % len(all_revs))
44318
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   620
    for r in range(len(index)):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   621
        if r not in all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   622
            msg = b"  revision missing from nodemap: %d\n" % r
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   623
            ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   624
            ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   625
        else:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   626
            all_revs.remove(r)
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 46089
diff changeset
   627
        nm_rev = _find_node(root, hex(index[r][7]))
44319
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   628
        if nm_rev is None:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   629
            msg = b"  revision node does not match any entries: %d\n" % r
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   630
            ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   631
            ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   632
        elif nm_rev != r:
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   633
            msg = (
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   634
                b"  revision node does not match the expected revision: "
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   635
                b"%d != %d\n" % (r, nm_rev)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   636
            )
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   637
            ui.write_err(msg)
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   638
            ret = 1
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   639
44318
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   640
    if all_revs:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   641
        for r in sorted(all_revs):
50510
4fe46f96b56b cli: fix spelling in `debugnodemap` error messages
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 49306
diff changeset
   642
            msg = b"  extra revisions in  nodemap: %d\n" % r
44318
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   643
            ui.write_err(msg)
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   644
        ret = 1
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   645
    return ret
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   646
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   647
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   648
def _all_revisions(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   649
    """return all revisions stored in a Trie"""
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   650
    for block in _walk_trie(root):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   651
        for v in block:
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   652
            if v is None or isinstance(v, Block):
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   653
                continue
20e125cdd719 nodemap: add basic checking of the on disk nodemap content
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44317
diff changeset
   654
            yield v
44319
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   655
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   656
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   657
def _find_node(block, node):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   658
    """find the revision associated with a given node"""
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   659
    entry = block.get(_to_int(node[0:1]))
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   660
    if isinstance(entry, dict):
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   661
        return _find_node(entry, node[1:])
d58206b70199 nodemap: all check that revision and nodes match in the nodemap
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44318
diff changeset
   662
    return entry
46469
835aafb2543f revlog: refactor logic to compute nodemap file in separate function
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46468
diff changeset
   663
835aafb2543f revlog: refactor logic to compute nodemap file in separate function
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46468
diff changeset
   664
47151
24be247a13b4 revlog: stop usage of `_indexfile` to computing nodemap path
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47148
diff changeset
   665
def get_nodemap_file(revlog):
47240
4f38ada3fc26 revlog: move the `trypending` logic from the `changelog` to the `revlog`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47164
diff changeset
   666
    if revlog._trypending:
47151
24be247a13b4 revlog: stop usage of `_indexfile` to computing nodemap path
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47148
diff changeset
   667
        pending_path = revlog.radix + b".n.a"
24be247a13b4 revlog: stop usage of `_indexfile` to computing nodemap path
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47148
diff changeset
   668
        if revlog.opener.exists(pending_path):
46469
835aafb2543f revlog: refactor logic to compute nodemap file in separate function
Pulkit Goyal <7895pulkit@gmail.com>
parents: 46468
diff changeset
   669
            return pending_path
47151
24be247a13b4 revlog: stop usage of `_indexfile` to computing nodemap path
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47148
diff changeset
   670
    return revlog.radix + b".n"