Mercurial Mercurial > mercurial / changeset
summary | shortlog | changelog | graph | tags | bookmarks | branches | files | changeset | raw | help
Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
store: introduce _matchtrackedpath() and use it to filter store files
authorPulkit Goyal <pulkit@yandex-team.ru>
Wed, 17 Oct 2018 17:42:32 +0300
changeset 40494 9aeb9e2d28a7
parent 40493 a2c4502e409b
child 40495 3a333a582d7b
store: introduce _matchtrackedpath() and use it to filter store files This patch introduces a function to filter store files on the basis of the path which they are tracking. The function assumes that the entries can be of two types, 'meta/*' and 'data/*' which means it will just work on revlog based storage and not with another storage ways. For the 'data/*' entries, we remove the 'data/' part and '.i/.d' part from the beginning and the end then pass that to matcher. For the 'meta/*' entries, we remove the 'meta/' and '/00manifest.(i/d)' part from beginning and end then call matcher.visitdir() with it to make sure all the parent directories are also downloaded. Since the storage filtering for narrow stream clones is implemented with this patch, we remove the un-implemented error message, add some more tests and add the treemanifest case to tests too. The tests demonstrate that it works correctly. After this patch, we have now narrow stream clones working. Narrow stream clones are a very important feature for large repositories who have good internet connection because they use streamclones for cloning and if they do normal narrow clone, that takes more time then a full streamclone. Also narrow-stream clone will drastically speed up clone timings. Differential Revision: https://phab.mercurial-scm.org/D5139
mercurial/store.py file | annotate | diff | comparison | revisions
mercurial/streamclone.py file | annotate | diff | comparison | revisions
tests/test-narrow-clone-stream.t file | annotate | diff | comparison | revisions 
--- a/mercurial/store.py	Wed Oct 17 17:24:55 2018 +0300
+++ b/mercurial/store.py	Wed Oct 17 17:42:32 2018 +0300
@@ -24,6 +24,20 @@
 
 parsers = policy.importmod(r'parsers')
 
+def _matchtrackedpath(path, matcher):
+    """parses a fncache entry and returns whether the entry is tracking a path
+    matched by matcher or not.
+
+    If matcher is None, returns True"""
+
+    if matcher is None:
+        return True
+    path = decodedir(path)
+    if path.startswith('data/'):
+        return matcher(path[len('data/'):-len('.i')])
+    elif path.startswith('meta/'):
+        return matcher.visitdir(path[len('meta/'):-len('/00manifest.i')] or '.')
+
 # This avoids a collision between a file named foo and a dir named
 # foo.i or foo.d
 def _encodedir(path):
@@ -413,6 +427,8 @@
 
     def datafiles(self, matcher=None):
         for a, b, size in super(encodedstore, self).datafiles():
+            if not _matchtrackedpath(a, matcher):
+                continue
             try:
                 a = decodefilename(a)
             except KeyError:
@@ -542,6 +558,8 @@
 
     def datafiles(self, matcher=None):
         for f in sorted(self.fncache):
+            if not _matchtrackedpath(f, matcher):
+                continue
             ef = self.encode(f)
             try:
                 yield f, ef, self.getsize(ef)
--- a/mercurial/streamclone.py	Wed Oct 17 17:24:55 2018 +0300
+++ b/mercurial/streamclone.py	Wed Oct 17 17:42:32 2018 +0300
@@ -545,10 +545,6 @@
     Returns a 3-tuple of (file count, file size, data iterator).
     """
 
-    # temporarily raise error until we add storage level logic
-    if includes or excludes:
-        raise error.Abort(_("server does not support narrow stream clones"))
-
     with repo.lock():
 
         entries = []
--- a/tests/test-narrow-clone-stream.t	Wed Oct 17 17:24:55 2018 +0300
+++ b/tests/test-narrow-clone-stream.t	Wed Oct 17 17:42:32 2018 +0300
@@ -1,7 +1,16 @@
+#testcases tree flat
+
 Tests narrow stream clones
 
   $ . "$TESTDIR/narrow-library.sh"
 
+#if tree
+  $ cat << EOF >> $HGRCPATH
+  > [experimental]
+  > treemanifest = 1
+  > EOF
+#endif
+
 Server setup
 
   $ hg init master
@@ -27,13 +36,51 @@
 
 Enable stream clone on the server
 
-  $ echo "[server]" >> master/.hg/hgrc
+  $ echo "[experimental.server]" >> master/.hg/hgrc
   $ echo "stream-narrow-clones=True" >> master/.hg/hgrc
 
 Cloning a specific file when stream clone is supported
 
   $ hg clone --narrow ssh://user@dummy/master narrow --noupdate --include "dir/src/f10" --stream
   streaming all changes
-  remote: abort: server does not support narrow stream clones
-  abort: pull failed on remote
-  [255]
+  * files to transfer, * KB of data (glob)
+  transferred * KB in * seconds (* */sec) (glob)
+
+  $ cd narrow
+  $ ls
+  $ hg tracked
+  I path:dir/src/f10
+
+Making sure we have the correct set of requirements
+
+  $ cat .hg/requires
+  dotencode
+  fncache
+  generaldelta
+  narrowhg-experimental
+  revlogv1
+  store
+  treemanifest (tree !)
+
+Making sure store has the required files
+
+  $ ls .hg/store/
+  00changelog.i
+  00manifest.i
+  data
+  fncache
+  meta (tree !)
+  narrowspec
+  undo
+  undo.backupfiles
+  undo.phaseroots
+
+Checking that repository has all the required data and not broken
+
+  $ hg verify
+  checking changesets
+  checking manifests
+  checking directory manifests (tree !)
+  crosschecking files in changesets and manifests
+  checking files
+  checked 40 changesets with 1 changes to 1 files
mercurial