tests/test-cache-abuse.t
author Jun Wu <quark@fb.com>
Fri, 02 Sep 2016 15:20:59 +0100
branchstable
changeset 29861 2f6d5c60f6fc
parent 29255 b0b85d8695cb
child 32478 1cc7c96cad75
permissions -rw-r--r--
annotate: pre-calculate the "needed" dictionary (issue5360) The "needed" dict is used as a reference counter to free items in the giant "hist" dict. However, currently it is not very accurate and can lead to dropping "hist" items unnecessarily, for example, with the following DAG, -3- / \ 0--1--2--4-- The current algorithm will visit and calculate rev 1 twice, undesired. And it tries to be smart by clearing rev 1's parents: "pcache[1] = []" at the time hist[1] being accessed (note: hist[1] needs to be used twice, by rev 2 and rev 3). It can result in incorrect results if p1 of rev 4 deletes chunks belonging to rev 0. However, simply removing "needed" is not okay, because it will consume 10x memory: # without any change % HGRCPATH= lrun ./hg annotate mercurial/commands.py -r d130a38 3>&2 [1] MEMORY 49074176 CPUTIME 9.213 REALTIME 9.270 # with "needed" removed MEMORY 637673472 CPUTIME 8.164 REALTIME 8.249 This patch moves "needed" (and "pcache") calculation to a separate DFS to address the issue. It improves perf and fixes issue5360 by correctly reusing hist, while maintaining low memory usage. Some additional attempt has been made to further reduce memory usage, like changing "pcache[f] = []" to "del pcache[f]". Therefore the result can be both faster and lower memory usage: # with this patch applied MEMORY 47575040 CPUTIME 7.870 REALTIME 7.926 [1]: lrun is a lightweight sandbox built on Linux cgroup and namespace. It's used to measure CPU and memory usage here. Source code is available at github.com/quark-zju/lrun.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
29041
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
     1
Enable obsolete markers
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
     2
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
     3
  $ cat >> $HGRCPATH << EOF
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
     4
  > [experimental]
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
     5
  > evolution=createmarkers
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
     6
  > [phases]
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
     7
  > publish=False
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
     8
  > EOF
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
     9
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    10
Build a repo with some cacheable bits:
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    11
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    12
  $ hg init a
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    13
  $ cd a
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    14
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    15
  $ echo a > a
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    16
  $ hg ci -qAm0
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    17
  $ hg tag t1
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    18
  $ hg book -i bk1
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    19
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    20
  $ hg branch -q b2
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    21
  $ hg ci -Am1
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    22
  $ hg tag t2
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    23
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    24
  $ echo dumb > dumb
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    25
  $ hg ci -qAmdumb
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    26
  $ hg debugobsolete b1174d11b69e63cb0c5726621a43c859f0858d7f
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    27
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    28
  $ hg phase -pr t1
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    29
  $ hg phase -fsr t2
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    30
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    31
Make a helper function to check cache damage invariants:
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    32
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    33
- command output shouldn't change
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    34
- cache should be present after first use
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    35
- corruption/repair should be silent (no exceptions or warnings)
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    36
- cache should survive deletion, overwrite, and append
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    37
- unreadable / unwriteable caches should be ignored
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    38
- cache should be rebuilt after corruption
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    39
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    40
  $ damage() {
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    41
  >  CMD=$1
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    42
  >  CACHE=.hg/cache/$2
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    43
  >  CLEAN=$3
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    44
  >  hg $CMD > before
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    45
  >  test -f $CACHE || echo "not present"
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    46
  >  echo bad > $CACHE
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    47
  >  test -z "$CLEAN" || $CLEAN
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    48
  >  hg $CMD > after
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    49
  >  diff -u before after || echo "*** overwrite corruption"
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    50
  >  echo corruption >> $CACHE
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    51
  >  test -z "$CLEAN" || $CLEAN
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    52
  >  hg $CMD > after
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    53
  >  diff -u before after || echo "*** append corruption"
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    54
  >  rm $CACHE
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    55
  >  mkdir $CACHE
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    56
  >  test -z "$CLEAN" || $CLEAN
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    57
  >  hg $CMD > after
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    58
  >  diff -u before after || echo "*** read-only corruption"
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    59
  >  test -d $CACHE || echo "*** directory clobbered"
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    60
  >  rmdir $CACHE
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    61
  >  test -z "$CLEAN" || $CLEAN
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    62
  >  hg $CMD > after
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    63
  >  diff -u before after || echo "*** missing corruption"
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    64
  >  test -f $CACHE || echo "not rebuilt"
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    65
  > }
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    66
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    67
Beat up tags caches:
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    68
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    69
  $ damage "tags --hidden" tags2
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    70
  $ damage tags tags2-visible
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    71
  $ damage "tag -f t3" hgtagsfnodes1
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    72
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    73
Beat up hidden cache:
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    74
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    75
  $ damage log hidden
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    76
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    77
Beat up branch caches:
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    78
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    79
  $ damage branches branch2-base "rm .hg/cache/branch2-[vs]*"
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    80
  $ damage branches branch2-served "rm .hg/cache/branch2-[bv]*"
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    81
  $ damage branches branch2-visible
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    82
  $ damage "log -r branch(.)" rbc-names-v1
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    83
  $ damage "log -r branch(default)" rbc-names-v1
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    84
  $ damage "log -r branch(b2)" rbc-revs-v1
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    85
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    86
We currently can't detect an rbc cache with unknown names:
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    87
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    88
  $ damage "log -qr branch(b2)" rbc-names-v1
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    89
  --- before	* (glob)
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    90
  +++ after	* (glob)
29255
b0b85d8695cb test-cache-abuse: correct for different hunk headers between Solaris and GNU
Danek Duvall <danek.duvall@oracle.com>
parents: 29041
diff changeset
    91
  @@ -1,8 +?,0 @@ (glob)
29041
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    92
  -2:5fb7d38b9dc4
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    93
  -3:60b597ffdafa
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    94
  -4:b1174d11b69e
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    95
  -5:6354685872c0
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    96
  -6:5ebc725f1bef
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    97
  -7:7b76eec2f273
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    98
  -8:ef3428d9d644
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
    99
  -9:ba7a936bc03c
b962ae0a0a05 tests: test a variety of cache invariants
Matt Mackall <mpm@selenic.com>
parents:
diff changeset
   100
  *** append corruption