lfs: introduce a localstore method for downloading from remote stores
The current local.write() method requires the full data, which means
concatenating file chunks in memory when downloading from a git server. The
dedicated method downloads in chunks, verifies the content on the fly, and
creates the usercache hardlink if successful. It can also be used for the file
system based remotestore.
An explicit division of labor between downloading from a remote store (which
should be verified) and writing to the store because of a commit or similar
(which doesn't need verification), seems clearer. I can't figure out how to
make a similar function for upload, because for a file remote store, it's a
simple open/read/write operation. For a gitremote store, it's open the file
and a urlreq.request(), and process that.
--- a/hgext/lfs/blobstore.py Tue Jan 02 12:14:08 2018 +0900
+++ b/hgext/lfs/blobstore.py Thu Dec 21 18:56:04 2017 -0500
@@ -114,6 +114,26 @@
return self.vfs(oid, 'rb')
+ def download(self, oid, src):
+ """Read the blob from the remote source in chunks, verify the content,
+ and write to this local blobstore."""
+ sha256 = hashlib.sha256()
+
+ with self.vfs(oid, 'wb', atomictemp=True) as fp:
+ for chunk in util.filechunkiter(src, size=1048576):
+ fp.write(chunk)
+ sha256.update(chunk)
+
+ realoid = sha256.hexdigest()
+ if realoid != oid:
+ raise error.Abort(_('corrupt remote lfs object: %s') % oid)
+
+ # XXX: should we verify the content of the cache, and hardlink back to
+ # the local store on success, but truncate, write and link on failure?
+ if not self.cachevfs.exists(oid):
+ self.ui.note(_('lfs: adding %s to the usercache\n') % oid)
+ lfutil.link(self.vfs.join(oid), self.cachevfs.join(oid))
+
def write(self, oid, data, verify=True):
"""Write blob to local blobstore."""
if verify: