branchcache: lazily validate nodes from the branchmap
On my personal hg-repository with 365 entries in .hg/cache/branch2, following
are the numbers for perfbranchmapload.
Before this patch:
! wall 0.000866 comb 0.000000 user 0.000000 sys 0.000000 (best of 2680)
! wall 0.001525 comb 0.000000 user 0.000000 sys 0.000000 (max of 2680)
! wall 0.001107 comb 0.001097 user 0.001086 sys 0.000011 (avg of 2680)
! wall 0.001104 comb 0.000000 user 0.000000 sys 0.000000 (median of 2680)
With this patch:
! wall 0.000530 comb 0.000000 user 0.000000 sys 0.000000 (best of 4240)
! wall 0.001078 comb 0.000000 user 0.000000 sys 0.000000 (max of 4240)
! wall 0.000696 comb 0.000693 user 0.000677 sys 0.000017 (avg of 4240)
! wall 0.000690 comb 0.000000 user 0.000000 sys 0.000000 (median of 4240)
On our internal repository with ~20k entries in branchcache, I see improvement
from 0.125 sec to 0.066 sec which is 47% speed up.
The above are the numbers of perfbranchmapload which shows how much time we
saved by not validating the nodes. But we need to validate some nodes. Following
are timings of some mercurial operations which have speed up because of this
lazy validation of nodes:
No-op `hg update` on our internal repository (Avg on 4 runs):
Before: 0.540 secs
After: 0.430 secs
Setting a branch name which already exists without --force (Avg of 4 runs):
Before: 0.510 secs
After: 0.250 secs
I ran the ASV performance suite and was unable to see any improvements except
there was improvement of perfdirstatewrite() on netbeans which I think was not
related.
I looked into the commit code, the command which I am trying to speedup, it
looks like it uses revbranchcache to update the branchcache.
Differential Revision: https://phab.mercurial-scm.org/D6208
$ . "$TESTDIR/narrow-library.sh"
initialize nested directories to validate complex include/exclude patterns
$ hg init master
$ cd master
$ cat >> .hg/hgrc <<EOF
> [narrow]
> serveellipses=True
> EOF
$ echo root > root
$ hg add root
$ hg commit -m 'add root'
$ for d in dir1 dir2 dir1/dirA dir1/dirB dir2/dirA dir2/dirB
> do
> mkdir -p $d
> echo $d/foo > $d/foo
> hg add $d/foo
> hg commit -m "add $d/foo"
> echo $d/bar > $d/bar
> hg add $d/bar
> hg commit -m "add $d/bar"
> done
#if execbit
$ chmod +x dir1/dirA/foo
$ hg commit -m "make dir1/dirA/foo executable"
#else
$ hg import --bypass - <<EOF
> # HG changeset patch
> make dir1/dirA/foo executable
>
> diff --git a/dir1/dirA/foo b/dir1/dirA/foo
> old mode 100644
> new mode 100755
> EOF
applying patch from stdin
$ hg update -qr tip
#endif
$ hg log -G -T '{rev} {node|short} {files}\n'
@ 13 c87ca422d521 dir1/dirA/foo
|
o 12 951b8a83924e dir2/dirB/bar
|
o 11 01ae5a51b563 dir2/dirB/foo
|
o 10 5eababdf0ac5 dir2/dirA/bar
|
o 9 99d690663739 dir2/dirA/foo
|
o 8 8e80155d5445 dir1/dirB/bar
|
o 7 406760310428 dir1/dirB/foo
|
o 6 623466a5f475 dir1/dirA/bar
|
o 5 06ff3a5be997 dir1/dirA/foo
|
o 4 33227af02764 dir2/bar
|
o 3 5e1f9d8d7c69 dir2/foo
|
o 2 594bc4b13d4a dir1/bar
|
o 1 47f480a08324 dir1/foo
|
o 0 2a4f0c3b67da root
$ cd ..
clone a narrow portion of the master, such that we can widen it later
$ hg clone --narrow ssh://user@dummy/master narrow \
> --include dir1 \
> --include dir2 \
> --exclude dir1/dirA \
> --exclude dir1/dirB \
> --exclude dir2/dirA \
> --exclude dir2/dirB
requesting all changes
adding changesets
adding manifests
adding file changes
added 6 changesets with 4 changes to 4 files
new changesets *:* (glob)
updating to branch default
4 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd narrow
$ hg tracked
I path:dir1
I path:dir2
X path:dir1/dirA
X path:dir1/dirB
X path:dir2/dirA
X path:dir2/dirB
$ hg manifest -r tip
dir1/bar
dir1/dirA/bar
dir1/dirA/foo
dir1/dirB/bar
dir1/dirB/foo
dir1/foo
dir2/bar
dir2/dirA/bar
dir2/dirA/foo
dir2/dirB/bar
dir2/dirB/foo
dir2/foo
root
$ find * | sort
dir1
dir1/bar
dir1/foo
dir2
dir2/bar
dir2/foo
$ hg log -G -T '{rev} {node|short}{if(ellipsis, "...")} {files}\n'
@ 5 c87ca422d521... dir1/dirA/foo
|
o 4 33227af02764 dir2/bar
|
o 3 5e1f9d8d7c69 dir2/foo
|
o 2 594bc4b13d4a dir1/bar
|
o 1 47f480a08324 dir1/foo
|
o 0 2a4f0c3b67da... root
widen the narrow checkout
$ hg tracked --removeexclude dir1/dirA
comparing with ssh://user@dummy/master
searching for changes
no changes found
saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-widen.hg (glob)
adding changesets
adding manifests
adding file changes
added 9 changesets with 6 changes to 6 files
new changesets *:* (glob)
$ hg tracked
I path:dir1
I path:dir2
X path:dir1/dirB
X path:dir2/dirA
X path:dir2/dirB
$ find * | sort
dir1
dir1/bar
dir1/dirA
dir1/dirA/bar
dir1/dirA/foo
dir1/foo
dir2
dir2/bar
dir2/foo
#if execbit
$ test -x dir1/dirA/foo && echo executable
executable
$ test -x dir1/dirA/bar || echo not executable
not executable
#endif
$ hg log -G -T '{rev} {node|short}{if(ellipsis, "...")} {files}\n'
@ 8 c87ca422d521 dir1/dirA/foo
|
o 7 951b8a83924e... dir2/dirB/bar
|
o 6 623466a5f475 dir1/dirA/bar
|
o 5 06ff3a5be997 dir1/dirA/foo
|
o 4 33227af02764 dir2/bar
|
o 3 5e1f9d8d7c69 dir2/foo
|
o 2 594bc4b13d4a dir1/bar
|
o 1 47f480a08324 dir1/foo
|
o 0 2a4f0c3b67da... root
widen narrow spec again, but exclude a file in previously included spec
$ hg tracked --removeexclude dir2/dirB --addexclude dir1/dirA/bar
comparing with ssh://user@dummy/master
searching for changes
looking for local changes to affected paths
deleting data/dir1/dirA/bar.i (reporevlogstore !)
deleting data/dir1/dirA/bar/0eca1d0cbdaea4651d1d04d71976a6d2d9bfaae5 (reposimplestore !)
deleting data/dir1/dirA/bar/index (reposimplestore !)
no changes found
saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-widen.hg (glob)
adding changesets
adding manifests
adding file changes
added 11 changesets with 7 changes to 7 files
new changesets *:* (glob)
$ hg tracked
I path:dir1
I path:dir2
X path:dir1/dirA/bar
X path:dir1/dirB
X path:dir2/dirA
$ find * | sort
dir1
dir1/bar
dir1/dirA
dir1/dirA/foo
dir1/foo
dir2
dir2/bar
dir2/dirB
dir2/dirB/bar
dir2/dirB/foo
dir2/foo
$ hg log -G -T '{rev} {node|short}{if(ellipsis, "...")} {files}\n'
@ 10 c87ca422d521 dir1/dirA/foo
|
o 9 951b8a83924e dir2/dirB/bar
|
o 8 01ae5a51b563 dir2/dirB/foo
|
o 7 5eababdf0ac5... dir2/dirA/bar
|
o 6 623466a5f475... dir1/dirA/bar
|
o 5 06ff3a5be997 dir1/dirA/foo
|
o 4 33227af02764 dir2/bar
|
o 3 5e1f9d8d7c69 dir2/foo
|
o 2 594bc4b13d4a dir1/bar
|
o 1 47f480a08324 dir1/foo
|
o 0 2a4f0c3b67da... root
widen narrow spec yet again, excluding a directory in previous spec
$ hg tracked --removeexclude dir2/dirA --addexclude dir1/dirA
comparing with ssh://user@dummy/master
searching for changes
looking for local changes to affected paths
deleting data/dir1/dirA/foo.i (reporevlogstore !)
deleting data/dir1/dirA/foo/162caeb3d55dceb1fee793aa631ac8c73fcb8b5e (reposimplestore !)
deleting data/dir1/dirA/foo/index (reposimplestore !)
no changes found
saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-widen.hg (glob)
adding changesets
adding manifests
adding file changes
added 13 changesets with 8 changes to 8 files
new changesets *:* (glob)
$ hg tracked
I path:dir1
I path:dir2
X path:dir1/dirA
X path:dir1/dirA/bar
X path:dir1/dirB
$ find * | sort
dir1
dir1/bar
dir1/foo
dir2
dir2/bar
dir2/dirA
dir2/dirA/bar
dir2/dirA/foo
dir2/dirB
dir2/dirB/bar
dir2/dirB/foo
dir2/foo
$ hg log -G -T '{rev} {node|short}{if(ellipsis, "...")} {files}\n'
@ 12 c87ca422d521... dir1/dirA/foo
|
o 11 951b8a83924e dir2/dirB/bar
|
o 10 01ae5a51b563 dir2/dirB/foo
|
o 9 5eababdf0ac5 dir2/dirA/bar
|
o 8 99d690663739 dir2/dirA/foo
|
o 7 8e80155d5445... dir1/dirB/bar
|
o 6 623466a5f475... dir1/dirA/bar
|
o 5 06ff3a5be997... dir1/dirA/foo
|
o 4 33227af02764 dir2/bar
|
o 3 5e1f9d8d7c69 dir2/foo
|
o 2 594bc4b13d4a dir1/bar
|
o 1 47f480a08324 dir1/foo
|
o 0 2a4f0c3b67da... root
include a directory that was previously explicitly excluded
$ hg tracked --removeexclude dir1/dirA
comparing with ssh://user@dummy/master
searching for changes
no changes found
saved backup bundle to $TESTTMP/narrow/.hg/strip-backup/*-widen.hg (glob)
adding changesets
adding manifests
adding file changes
added 13 changesets with 9 changes to 9 files
new changesets *:* (glob)
$ hg tracked
I path:dir1
I path:dir2
X path:dir1/dirA/bar
X path:dir1/dirB
$ find * | sort
dir1
dir1/bar
dir1/dirA
dir1/dirA/foo
dir1/foo
dir2
dir2/bar
dir2/dirA
dir2/dirA/bar
dir2/dirA/foo
dir2/dirB
dir2/dirB/bar
dir2/dirB/foo
dir2/foo
$ hg log -G -T '{rev} {node|short}{if(ellipsis, "...")} {files}\n'
@ 12 c87ca422d521 dir1/dirA/foo
|
o 11 951b8a83924e dir2/dirB/bar
|
o 10 01ae5a51b563 dir2/dirB/foo
|
o 9 5eababdf0ac5 dir2/dirA/bar
|
o 8 99d690663739 dir2/dirA/foo
|
o 7 8e80155d5445... dir1/dirB/bar
|
o 6 623466a5f475... dir1/dirA/bar
|
o 5 06ff3a5be997 dir1/dirA/foo
|
o 4 33227af02764 dir2/bar
|
o 3 5e1f9d8d7c69 dir2/foo
|
o 2 594bc4b13d4a dir1/bar
|
o 1 47f480a08324 dir1/foo
|
o 0 2a4f0c3b67da... root
$ cd ..
clone a narrow portion of the master, such that we can widen it later
$ hg clone --narrow ssh://user@dummy/master narrow2 --include dir1/dirA
requesting all changes
adding changesets
adding manifests
adding file changes
added 5 changesets with 2 changes to 2 files
new changesets *:* (glob)
updating to branch default
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd narrow2
$ find * | sort
dir1
dir1/dirA
dir1/dirA/bar
dir1/dirA/foo
$ hg tracked --addinclude dir1
comparing with ssh://user@dummy/master
searching for changes
no changes found
saved backup bundle to $TESTTMP/narrow2/.hg/strip-backup/*-widen.hg (glob)
adding changesets
adding manifests
adding file changes
added 10 changesets with 6 changes to 6 files
new changesets *:* (glob)
$ find * | sort
dir1
dir1/bar
dir1/dirA
dir1/dirA/bar
dir1/dirA/foo
dir1/dirB
dir1/dirB/bar
dir1/dirB/foo
dir1/foo
$ hg log -G -T '{rev} {node|short}{if(ellipsis, "...")} {files}\n'
@ 9 c87ca422d521 dir1/dirA/foo
|
o 8 951b8a83924e... dir2/dirB/bar
|
o 7 8e80155d5445 dir1/dirB/bar
|
o 6 406760310428 dir1/dirB/foo
|
o 5 623466a5f475 dir1/dirA/bar
|
o 4 06ff3a5be997 dir1/dirA/foo
|
o 3 33227af02764... dir2/bar
|
o 2 594bc4b13d4a dir1/bar
|
o 1 47f480a08324 dir1/foo
|
o 0 2a4f0c3b67da... root
Illegal patterns are rejected
$ hg tracked --addinclude glob:**
abort: invalid prefix on narrow pattern: glob:**
(narrow patterns must begin with one of the following: path:, rootfilesin:)
[255]
$ hg tracked --addexclude set:ignored
abort: invalid prefix on narrow pattern: set:ignored
(narrow patterns must begin with one of the following: path:, rootfilesin:)
[255]
$ cat .hg/store/narrowspec
[include]
path:dir1
path:dir1/dirA
[exclude]
$ cat > .hg/store/narrowspec << EOF
> [include]
> glob:**
> EOF
$ hg tracked
abort: invalid prefix on narrow pattern: glob:**
(narrow patterns must begin with one of the following: path:, rootfilesin:)
[255]
$ cat > .hg/store/narrowspec << EOF
> [include]
> path:.
> [exclude]
> set:ignored
> EOF
$ hg tracked
abort: invalid prefix on narrow pattern: set:ignored
(narrow patterns must begin with one of the following: path:, rootfilesin:)
[255]