manifest: persist the manifestfulltext cache Reconstructing the manifest from the revlog takes time, so much so that there already is a LRU cache to avoid having to load a manifest multiple times. This patch persists that LRU cache in the .hg/cache directory, so we can re-use this cache across hg commands. Commit benchmark (run on Macos 10.13 on a 2017-model Macbook Pro with Core i7 2.9GHz and flash drive), testing without and with patch run 5 times, baseline is r2a227782e754: * committing to an existing file, against the mozilla-central repository. Baseline real time average 1.9692, with patch 1.3786. A new debugcommand "hg debugmanifestfulltextcache" lets you inspect the cache, clear it, or add specific manifest nodeids to it. When calling repo.updatecaches(), the manifest(s) for the working copy parents are added to the cache. The hg perfmanifest command has an additional --clear-disk switch to clear this cache when testing manifest loading performance. Using this command to test performance on the firefox repository for revision f947d902ed91, whose manifest has a delta chain length of 60540, we see: $ hg perfmanifest f947d902ed91 --clear-disk ! wall 0.972253 comb 0.970000 user 0.850000 sys 0.120000 (best of 10) $ hg debugmanifestfulltextcache -a `hg log --debug -r f947d902ed91 | grep manifest | cut -d: -f3` Cache contains 1 manifest entries, in order of most to least recent: id: 0294517df4aad07c70701db43bc7ff24c3ce7dbc, size 25.6 MB Total cache data size 25.6 MB, on-disk 0 bytes $ hg perfmanifest f947d902ed91 ! wall 0.036748 comb 0.040000 user 0.020000 sys 0.020000 (best of 100) Worst-case scenario: a manifest text loaded from a single delta; in the firefox repository manifest node 9a1246ff762e is the chain base for the manifest attached to revision f947d902ed91. Loading this from a full cache file is just as fast as without the cache; the extra node ids ensure a big full cache: $ for node in 9a1246ff762e 1a1922c14a3e 54a31d11a36a 0294517df4aa; do > hgd debugmanifestfulltextcache -a $node > /dev/null > done $ hgd perfmanifest -m 9a1246ff762e ! wall 0.077513 comb 0.080000 user 0.030000 sys 0.050000 (best of 100) $ hgd perfmanifest -m 9a1246ff762e --clear-disk ! wall 0.078547 comb 0.080000 user 0.070000 sys 0.010000 (best of 100)

     1

Corrupt an hg repo with two pulls.

     2

create one repo with a long history

     3

     4

  $ hg init source1

     5

  $ cd source1

     6

  $ touch foo

     7

  $ hg add foo

     8

  $ for i in 1 2 3 4 5 6 7 8 9 10; do

     9

  >     echo $i >> foo

    10

  >     hg ci -m $i

    11

  > done

    12

  $ cd ..

    13

    14

create one repo with a shorter history

    15

    16

  $ hg clone -r 0 source1 source2

    17

  adding changesets

    18

  adding manifests

    19

  adding file changes

    20

  added 1 changesets with 1 changes to 1 files

    21

  new changesets 495a0ec48aaf

    22

  updating to branch default

    23

  1 files updated, 0 files merged, 0 files removed, 0 files unresolved

    24

  $ cd source2

    25

  $ echo a >> foo

    26

  $ hg ci -m a

    27

  $ cd ..

    28

    29

create a third repo to pull both other repos into it

    30

    31

  $ hg init corrupted

    32

  $ cd corrupted

    33

    34

use a hook to make the second pull start while the first one is still running

    35

    36

  $ echo '[hooks]' >> .hg/hgrc

    37

  $ echo 'prechangegroup = sleep 5' >> .hg/hgrc

    38

    39

start a pull...

    40

    41

  $ hg pull ../source1 > pull.out 2>&1 &

    42

    43

... and start another pull before the first one has finished

    44

    45

  $ sleep 1

    46

  $ hg pull ../source2 2>/dev/null

    47

  pulling from ../source2

    48

  searching for changes

    49

  adding changesets

    50

  adding manifests

    51

  adding file changes

    52

  added 1 changesets with 1 changes to 1 files (+1 heads)

    53

  new changesets ca3c05af513e

    54

  (run 'hg heads' to see heads, 'hg merge' to merge)

    55

  $ cat pull.out

    56

  pulling from ../source1

    57

  requesting all changes

    58

  adding changesets

    59

  adding manifests

    60

  adding file changes

    61

  added 10 changesets with 10 changes to 1 files

    62

  new changesets 495a0ec48aaf:1e7b6c812ca8

    63

  (run 'hg update' to get a working copy)

    64

    65

see the result

    66

    67

  $ wait

    68

  $ hg verify

    69

  checking changesets

    70

  checking manifests

    71

  crosschecking files in changesets and manifests

    72

  checking files

    73

  1 files, 11 changesets, 11 total revisions

    74

    75

  $ cd ..