hgext/largefiles/basestore.py
author Manuel Jacob <me@manueljacob.de>
Mon, 11 Jul 2022 01:51:20 +0200
branchstable
changeset 49378 094a5fa3cf52
parent 48946 642e31cb55f0
permissions -rw-r--r--
procutil: make stream detection in make_line_buffered more correct and strict In make_line_buffered(), we don’t want to wrap the stream if we know that lines get flushed to the underlying raw stream already. Previously, the heuristic was too optimistic. It assumed that any stream which is not an instance of io.BufferedIOBase doesn’t need wrapping. However, there are buffered streams that aren’t instances of io.BufferedIOBase, like Mercurial’s own winstdout. The new logic is different in two ways: First, only for the check, if unwraps any combination of WriteAllWrapper and winstdout. Second, it skips wrapping the stream only if it is an instance of io.RawIOBase (or already wrapped). If it is an instance of io.BufferedIOBase, it gets wrapped. In any other case, the function raises an exception. This ensures that, if an unknown stream is passed or we add another wrapper in the future, we don’t wrap the stream if it’s already line buffered or not wrap the stream if it’s not line buffered. In fact, this was already helpful during development of this change. Without it, I possibly would have forgot that WriteAllWrapper needs to be ignored for the check, leading to unnecessary wrapping if stdout is unbuffered. The alternative would have been to always wrap unknown streams. However, I don’t think that anyone would benefit from being less strict. We can expect streams from the standard library to be subclassing either io.RawIOBase or io.BufferedIOBase, so running Mercurial in the standard way should not regress by this change. Py2exe might replace sys.stdout and sys.stderr, but that currently breaks Mercurial anyway and also these streams don’t claim to be interactive, so this function is not called for them.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     1
# Copyright 2009-2010 Gregory P. Ward
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     2
# Copyright 2009-2010 Intelerad Medical Systems Incorporated
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     3
# Copyright 2010-2011 Fog Creek Software
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     4
# Copyright 2010-2011 Unity Technologies
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     5
#
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     6
# This software may be used and distributed according to the terms of the
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     7
# GNU General Public License version 2 or any later version.
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
     8
15252
6e809bb4f969 largefiles: improve comments, internal docstrings
Greg Ward <greg@gerg.ca>
parents: 15169
diff changeset
     9
'''base class for store implementations and store-related utility code'''
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    10
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    11
from mercurial.i18n import _
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    12
47012
d55b71393907 node: replace nullid and friends with nodeconstants class
Joerg Sonnenberger <joerg@bec.de>
parents: 46907
diff changeset
    13
from mercurial.node import short
d55b71393907 node: replace nullid and friends with nodeconstants class
Joerg Sonnenberger <joerg@bec.de>
parents: 46907
diff changeset
    14
from mercurial import util
46907
ffd3e823a7e5 urlutil: extract `url` related code from `util` into the new module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 45942
diff changeset
    15
from mercurial.utils import (
ffd3e823a7e5 urlutil: extract `url` related code from `util` into the new module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 45942
diff changeset
    16
    urlutil,
ffd3e823a7e5 urlutil: extract `url` related code from `util` into the new module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 45942
diff changeset
    17
)
29307
67999697221a py3: make largefiles/basestore.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 29305
diff changeset
    18
67999697221a py3: make largefiles/basestore.py use absolute_import
liscju <piotr.listkiewicz@gmail.com>
parents: 29305
diff changeset
    19
from . import lfutil
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    20
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
    21
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    22
class StoreError(Exception):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44452
diff changeset
    23
    """Raised when there is a problem getting files from or putting
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44452
diff changeset
    24
    files to a central store."""
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
    25
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    26
    def __init__(self, filename, hash, url, detail):
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    27
        self.filename = filename
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    28
        self.hash = hash
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    29
        self.url = url
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    30
        self.detail = detail
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    31
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    32
    def longmessage(self):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
    33
        return _(b"error getting id %s from url %s for file %s: %s\n") % (
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
    34
            self.hash,
46907
ffd3e823a7e5 urlutil: extract `url` related code from `util` into the new module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 45942
diff changeset
    35
            urlutil.hidepassword(self.url),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
    36
            self.filename,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
    37
            self.detail,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
    38
        )
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    39
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    40
    def __str__(self):
46907
ffd3e823a7e5 urlutil: extract `url` related code from `util` into the new module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 45942
diff changeset
    41
        return b"%s: %s" % (urlutil.hidepassword(self.url), self.detail)
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    42
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
    43
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48875
diff changeset
    44
class basestore:
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    45
    def __init__(self, ui, repo, url):
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    46
        self.ui = ui
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    47
        self.repo = repo
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    48
        self.url = url
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    49
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    50
    def put(self, source, hash):
19007
266b5fb72f26 largefiles: 'put' should store 'source' file in under 'hash', also in localstore
Mads Kiilerich <madski@unity3d.com>
parents: 19003
diff changeset
    51
        '''Put source file into the store so it can be retrieved by hash.'''
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
    52
        raise NotImplementedError(b'abstract method')
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    53
17127
9e1616307c4c largefiles: batch statlfile requests when pushing a largefiles repo (issue3386)
Na'Tosha Bard <natosha@unity3d.com>
parents: 16247
diff changeset
    54
    def exists(self, hashes):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44452
diff changeset
    55
        """Check to see if the store contains the given hashes. Given an
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44452
diff changeset
    56
        iterable of hashes it returns a mapping from hash to bool."""
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
    57
        raise NotImplementedError(b'abstract method')
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    58
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    59
    def get(self, files):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44452
diff changeset
    60
        """Get the specified largefiles from the store and write to local
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    61
        files under repo.root.  files is a list of (filename, hash)
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 17127
diff changeset
    62
        tuples.  Return (success, missing), lists of files successfully
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    63
        downloaded and those not found in the store.  success is a list
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    64
        of (filename, hash) tuples; missing is a list of filenames that
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    65
        we could not get.  (The detailed error message will already have
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    66
        been presented to the user, so missing is just supplied as a
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44452
diff changeset
    67
        summary.)"""
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    68
        success = []
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    69
        missing = []
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    70
        ui = self.ui
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    71
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    72
        at = 0
44452
9d2b2df2c2ba cleanup: run pyupgrade on our source tree to clean up varying things
Augie Fackler <augie@google.com>
parents: 43077
diff changeset
    73
        available = self.exists({hash for (_filename, hash) in files})
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
    74
        with ui.makeprogress(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
    75
            _(b'getting largefiles'), unit=_(b'files'), total=len(files)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
    76
        ) as progress:
39390
a65ad9b22a00 largefiles: use a context manager to control the progress bar lifetime
Matt Harbison <matt_harbison@yahoo.com>
parents: 38407
diff changeset
    77
            for filename, hash in files:
a65ad9b22a00 largefiles: use a context manager to control the progress bar lifetime
Matt Harbison <matt_harbison@yahoo.com>
parents: 38407
diff changeset
    78
                progress.update(at)
a65ad9b22a00 largefiles: use a context manager to control the progress bar lifetime
Matt Harbison <matt_harbison@yahoo.com>
parents: 38407
diff changeset
    79
                at += 1
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
    80
                ui.note(_(b'getting %s:%s\n') % (filename, hash))
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    81
39390
a65ad9b22a00 largefiles: use a context manager to control the progress bar lifetime
Matt Harbison <matt_harbison@yahoo.com>
parents: 38407
diff changeset
    82
                if not available.get(hash):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
    83
                    ui.warn(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
    84
                        _(b'%s: largefile %s not available from %s\n')
46907
ffd3e823a7e5 urlutil: extract `url` related code from `util` into the new module
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 45942
diff changeset
    85
                        % (filename, hash, urlutil.hidepassword(self.url))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
    86
                    )
39390
a65ad9b22a00 largefiles: use a context manager to control the progress bar lifetime
Matt Harbison <matt_harbison@yahoo.com>
parents: 38407
diff changeset
    87
                    missing.append(filename)
a65ad9b22a00 largefiles: use a context manager to control the progress bar lifetime
Matt Harbison <matt_harbison@yahoo.com>
parents: 38407
diff changeset
    88
                    continue
19008
9d33d6e0d442 largefiles: stat all largefiles in one batch before downloading
Mads Kiilerich <madski@unity3d.com>
parents: 19007
diff changeset
    89
39390
a65ad9b22a00 largefiles: use a context manager to control the progress bar lifetime
Matt Harbison <matt_harbison@yahoo.com>
parents: 38407
diff changeset
    90
                if self._gethash(filename, hash):
a65ad9b22a00 largefiles: use a context manager to control the progress bar lifetime
Matt Harbison <matt_harbison@yahoo.com>
parents: 38407
diff changeset
    91
                    success.append((filename, hash))
a65ad9b22a00 largefiles: use a context manager to control the progress bar lifetime
Matt Harbison <matt_harbison@yahoo.com>
parents: 38407
diff changeset
    92
                else:
a65ad9b22a00 largefiles: use a context manager to control the progress bar lifetime
Matt Harbison <matt_harbison@yahoo.com>
parents: 38407
diff changeset
    93
                    missing.append(filename)
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    94
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    95
        return (success, missing)
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
    96
19918
ae65192fd6b4 largefiles: refactor basestore, extract _gethash method
Mads Kiilerich <madski@unity3d.com>
parents: 19008
diff changeset
    97
    def _gethash(self, filename, hash):
ae65192fd6b4 largefiles: refactor basestore, extract _gethash method
Mads Kiilerich <madski@unity3d.com>
parents: 19008
diff changeset
    98
        """Get file with the provided hash and store it in the local repo's
ae65192fd6b4 largefiles: refactor basestore, extract _gethash method
Mads Kiilerich <madski@unity3d.com>
parents: 19008
diff changeset
    99
        store and in the usercache.
ae65192fd6b4 largefiles: refactor basestore, extract _gethash method
Mads Kiilerich <madski@unity3d.com>
parents: 19008
diff changeset
   100
        filename is for informational messages only.
ae65192fd6b4 largefiles: refactor basestore, extract _gethash method
Mads Kiilerich <madski@unity3d.com>
parents: 19008
diff changeset
   101
        """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   102
        util.makedirs(lfutil.storepath(self.repo, b''))
19918
ae65192fd6b4 largefiles: refactor basestore, extract _gethash method
Mads Kiilerich <madski@unity3d.com>
parents: 19008
diff changeset
   103
        storefilename = lfutil.storepath(self.repo, hash)
ae65192fd6b4 largefiles: refactor basestore, extract _gethash method
Mads Kiilerich <madski@unity3d.com>
parents: 19008
diff changeset
   104
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   105
        tmpname = storefilename + b'.tmp'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
   106
        with util.atomictempfile(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
   107
            tmpname, createmode=self.repo.store.createmode
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
   108
        ) as tmpfile:
30142
3dcaf1c4e90d largefiles: use context for file closing
Mads Kiilerich <madski@unity3d.com>
parents: 29307
diff changeset
   109
            try:
3dcaf1c4e90d largefiles: use context for file closing
Mads Kiilerich <madski@unity3d.com>
parents: 29307
diff changeset
   110
                gothash = self._getfile(tmpfile, filename, hash)
3dcaf1c4e90d largefiles: use context for file closing
Mads Kiilerich <madski@unity3d.com>
parents: 29307
diff changeset
   111
            except StoreError as err:
3dcaf1c4e90d largefiles: use context for file closing
Mads Kiilerich <madski@unity3d.com>
parents: 29307
diff changeset
   112
                self.ui.warn(err.longmessage())
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   113
                gothash = b""
19918
ae65192fd6b4 largefiles: refactor basestore, extract _gethash method
Mads Kiilerich <madski@unity3d.com>
parents: 19008
diff changeset
   114
ae65192fd6b4 largefiles: refactor basestore, extract _gethash method
Mads Kiilerich <madski@unity3d.com>
parents: 19008
diff changeset
   115
        if gothash != hash:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   116
            if gothash != b"":
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
   117
                self.ui.warn(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   118
                    _(b'%s: data corruption (expected %s, got %s)\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
   119
                    % (filename, hash, gothash)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
   120
                )
19918
ae65192fd6b4 largefiles: refactor basestore, extract _gethash method
Mads Kiilerich <madski@unity3d.com>
parents: 19008
diff changeset
   121
            util.unlink(tmpname)
ae65192fd6b4 largefiles: refactor basestore, extract _gethash method
Mads Kiilerich <madski@unity3d.com>
parents: 19008
diff changeset
   122
            return False
ae65192fd6b4 largefiles: refactor basestore, extract _gethash method
Mads Kiilerich <madski@unity3d.com>
parents: 19008
diff changeset
   123
ae65192fd6b4 largefiles: refactor basestore, extract _gethash method
Mads Kiilerich <madski@unity3d.com>
parents: 19008
diff changeset
   124
        util.rename(tmpname, storefilename)
ae65192fd6b4 largefiles: refactor basestore, extract _gethash method
Mads Kiilerich <madski@unity3d.com>
parents: 19008
diff changeset
   125
        lfutil.linktousercache(self.repo, hash)
ae65192fd6b4 largefiles: refactor basestore, extract _gethash method
Mads Kiilerich <madski@unity3d.com>
parents: 19008
diff changeset
   126
        return True
ae65192fd6b4 largefiles: refactor basestore, extract _gethash method
Mads Kiilerich <madski@unity3d.com>
parents: 19008
diff changeset
   127
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   128
    def verify(self, revs, contents=False):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44452
diff changeset
   129
        """Verify the existence (and, optionally, contents) of every big
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   130
        file revision referenced by every changeset in revs.
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44452
diff changeset
   131
        Return 0 if all is well, non-zero on any errors."""
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   132
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
   133
        self.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   134
            _(b'searching %d changesets for largefiles\n') % len(revs)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
   135
        )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
   136
        verified = set()  # set of (filename, filenode) tuples
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
   137
        filestocheck = []  # list of (cset, filename, expectedhash)
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   138
        for rev in revs:
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   139
            cctx = self.repo[rev]
47012
d55b71393907 node: replace nullid and friends with nodeconstants class
Joerg Sonnenberger <joerg@bec.de>
parents: 46907
diff changeset
   140
            cset = b"%d:%s" % (cctx.rev(), short(cctx.node()))
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   141
18486
1067a6240f86 largefiles: verify all files in each revision and report errors in any revision
Mads Kiilerich <madski@unity3d.com>
parents: 18483
diff changeset
   142
            for standin in cctx:
29067
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 28463
diff changeset
   143
                filename = lfutil.splitstandin(standin)
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 28463
diff changeset
   144
                if filename:
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 28463
diff changeset
   145
                    fctx = cctx[standin]
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 28463
diff changeset
   146
                    key = (filename, fctx.filenode())
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 28463
diff changeset
   147
                    if key not in verified:
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 28463
diff changeset
   148
                        verified.add(key)
31740
a40e979b9d97 largefiles: use readasstandin() to read hex hash directly from filectx
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 31658
diff changeset
   149
                        expectedhash = lfutil.readasstandin(fctx)
29067
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 28463
diff changeset
   150
                        filestocheck.append((cset, filename, expectedhash))
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 28463
diff changeset
   151
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 28463
diff changeset
   152
        failed = self._verifyfiles(contents, filestocheck)
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   153
16247
d87d9d8a8e03 largefiles: remove use of underscores that breaks coding convention
Na'Tosha Bard <natosha@unity3d.com>
parents: 16154
diff changeset
   154
        numrevs = len(verified)
42057
566daffc607d cleanup: use set literals where possible
Martin von Zweigbergk <martinvonz@google.com>
parents: 39390
diff changeset
   155
        numlfiles = len({fname for (fname, fnode) in verified})
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   156
        if contents:
18546
fb0e8966a4be largefiles: verify status should be written as status, not as write
Mads Kiilerich <madski@unity3d.com>
parents: 18489
diff changeset
   157
            self.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   158
                _(b'verified contents of %d revisions of %d largefiles\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
   159
                % (numrevs, numlfiles)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
   160
            )
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   161
        else:
18546
fb0e8966a4be largefiles: verify status should be written as status, not as write
Mads Kiilerich <madski@unity3d.com>
parents: 18489
diff changeset
   162
            self.ui.status(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   163
                _(b'verified existence of %d revisions of %d largefiles\n')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
   164
                % (numrevs, numlfiles)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42057
diff changeset
   165
            )
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   166
        return int(failed)
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   167
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   168
    def _getfile(self, tmpfile, filename, hash):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44452
diff changeset
   169
        """Fetch one revision of one file from the store and write it
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   170
        to tmpfile.  Compute the hash of the file on-the-fly as it
18999
c1b5f9c4d989 largefiles: refactoring - return hex from _getfile and copyandhash
Mads Kiilerich <madski@unity3d.com>
parents: 18731
diff changeset
   171
        downloads and return the hash.  Close tmpfile.  Raise
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   172
        StoreError if unable to download the file (e.g. it does not
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44452
diff changeset
   173
        exist in the store)."""
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   174
        raise NotImplementedError(b'abstract method')
15168
cfccd3bee7b3 hgext: add largefiles extension
various
parents:
diff changeset
   175
29067
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 28463
diff changeset
   176
    def _verifyfiles(self, contents, filestocheck):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44452
diff changeset
   177
        """Perform the actual verification of files in the store.
18574
4db9e31ae605 largefiles: docstrings for verify methods
Mads Kiilerich <mads@kiilerich.com>
parents: 18573
diff changeset
   178
        'contents' controls verification of content hash.
29067
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 28463
diff changeset
   179
        'filestocheck' is list of files to check.
207c0db08953 largefiles: change basestore._verifyfile to take list of files to check
liscju <piotr.listkiewicz@gmail.com>
parents: 28463
diff changeset
   180
        Returns _true_ if any problems are found!
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 44452
diff changeset
   181
        """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   182
        raise NotImplementedError(b'abstract method')