tests/test-clone-stream-revlog-split.t
author Pierre-Yves David <pierre-yves.david@octobus.net>
Fri, 05 Apr 2024 11:05:54 +0200
changeset 51576 de5bf3fe0233
parent 51181 dcaa2df1f688
permissions -rw-r--r--
revset: stop serializing node when using "%ln" Turning hundred of thousand of node from node to hex and back can be slow… what about we stop doing it? In many case were we are using node id we should be using revision id. However this is not a good reason to have a stupidly slow implementation of "%ln". This caught my attention again because the phase discovery during push make an extensive use of "%ln" or huge set. In absolute, that phase discovery probably should use "%ld" and need to improves its algorithmic complexity, but improving "%ln" seems simple and long overdue. This greatly speeds up `hg push` on repository with many drafts. Here are some relevant poulpe benchmarks: ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # benchmark.name = hg.command.push # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.explicit-rev = all-out-heads # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = default ## benchmark.variants.revs = any-1-extra-rev before: 44.235070 after: 20.416329 (-53.85%, -23.82) ## benchmark.variants.revs = any-100-extra-rev before: 49.234697 after: 26.519829 (-46.14%, -22.71) ### benchmark.name = hg.command.bundle # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.revs = all # benchmark.variants.type = none-streamv2 ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog before: 10.138396 after: 7.750458 (-23.55%, -2.39) ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog before: 1.263859 after: 0.700229 (-44.60%, -0.56) ## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog before: 399.484481 after: 346.5089 (-13.26%, -52.98) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog before: 4.540080 after: 3.401700 (-25.07%, -1.14) ## data-env-vars.name = tryton-public-2024-03-22-zstd-sparse-revlog before: 2.975765 after: 1.870798 (-37.13%, -1.10)

Test stream cloning while a revlog split happens
------------------------------------------------

#testcases stream-bundle2-v2 stream-bundle2-v3

#if stream-bundle2-v3
  $ cat << EOF >> $HGRCPATH
  > [experimental]
  > stream-v3 = yes
  > EOF
#endif

setup a repository for tests
----------------------------

  $ cat >> $HGRCPATH << EOF
  > [format]
  > # skip compression to make it easy to trigger a split
  > revlog-compression=none
  > [phases]
  > publish=no
  > EOF

  $ hg init server
  $ cd server
  $ file="some-file"
  $ printf '%20d' '1' > $file
  $ hg commit -Aqma
  $ printf '%1024d' '1' > $file
  $ hg commit -Aqmb
  $ printf '%20d' '1' > $file
  $ hg commit -Aqmc

check the revlog is inline

  $ f -s .hg/store/data/some-file*
  .hg/store/data/some-file.i: size=1259
  $ hg debug-revlog-index some-file
     rev linkrev       nodeid    p1-nodeid    p2-nodeid
       0       0 ed70cecbc103 000000000000 000000000000
       1       1 7241018db64c ed70cecbc103 000000000000
       2       2 fa1120531cc1 7241018db64c 000000000000
  $ cd ..

setup synchronisation file

  $ HG_TEST_STREAM_WALKED_FILE_1="$TESTTMP/sync_file_walked_1"
  $ export HG_TEST_STREAM_WALKED_FILE_1
  $ HG_TEST_STREAM_WALKED_FILE_2="$TESTTMP/sync_file_walked_2"
  $ export HG_TEST_STREAM_WALKED_FILE_2
  $ HG_TEST_STREAM_WALKED_FILE_3="$TESTTMP/sync_file_walked_3"
  $ export HG_TEST_STREAM_WALKED_FILE_3


Test stream-clone raced by a revlog-split
=========================================

Test stream-clone where the file is split right after the lock section is done

Start the server

  $ hg serve -R server \
  >   -p $HGPORT1 -d --error errors.log --pid-file=hg.pid \
  >   --config extensions.stream_steps="$RUNTESTDIR/testlib/ext-stream-clone-steps.py"
  $ cat hg.pid >> $DAEMON_PIDS

Start a client doing a streaming clone

  $ ( \
  >    hg clone --debug --stream -U http://localhost:$HGPORT1 \
  >    clone-while-split > client.log 2>&1; \
  >    touch "$HG_TEST_STREAM_WALKED_FILE_3" \
  > ) &

Wait for the server to be done collecting data

  $ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_1

trigger a split

  $ dd if=/dev/zero of=server/$file bs=1k count=128 > /dev/null 2>&1
  $ hg -R server ci -m "triggering a split" --config ui.timeout.warn=-1

unlock the stream generation

  $ touch $HG_TEST_STREAM_WALKED_FILE_2

wait for the client to be done cloning.

  $ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_3

Check everything is fine

  $ cat client.log
  using http://localhost:$HGPORT1/
  sending capabilities command
  query 1; heads
  sending batch command
  streaming all changes
  sending getbundle command
  bundle2-input-bundle: with-transaction
  bundle2-input-part: "stream2" (params: 3 mandatory) supported (stream-bundle2-v2 !)
  bundle2-input-part: "stream3-exp" (params: 1 mandatory) supported (stream-bundle2-v3 !)
  applying stream bundle
  8 files to transfer, 2.11 KB of data (stream-bundle2-v2 no-rust !)
  10 files to transfer, 2.29 KB of data (stream-bundle2-v2 rust !)
  adding [s] data/some-file.i (1.23 KB) (stream-bundle2-v2 !)
  7 entries to transfer (stream-bundle2-v3 !)
  adding [s] data/some-file.d (1.04 KB) (stream-bundle2-v3 !)
  adding [s] data/some-file.i (192 bytes) (stream-bundle2-v3 !)
  adding [s] phaseroots (43 bytes)
  adding [s] 00manifest.i (348 bytes)
  adding [s] 00changelog.n (62 bytes) (rust !)
  adding [s] 00changelog-88698448.nd (128 bytes) (rust !)
  adding [s] 00changelog.d (189 bytes)
  adding [s] 00changelog.i (192 bytes)
  adding [c] branch2-served (94 bytes)
  adding [c] rbc-names-v1 (7 bytes)
  adding [c] rbc-revs-v1 (24 bytes)
  updating the branch cache
  transferred 2.11 KB in * seconds (* */sec) (glob) (no-rust !)
  transferred 2.29 KB in * seconds (* */sec) (glob) (rust !)
  bundle2-input-part: total payload size 2285 (stream-bundle2-v2 no-rust !)
  bundle2-input-part: total payload size 2518 (stream-bundle2-v2 rust !)
  bundle2-input-part: total payload size 2313 (stream-bundle2-v3 no-rust !)
  bundle2-input-part: total payload size 2546 (stream-bundle2-v3 rust !)
  bundle2-input-part: "listkeys" (params: 1 mandatory) supported
  bundle2-input-bundle: 2 parts total
  checking for updated bookmarks
  updating the branch cache
  (sent 3 HTTP requests and * bytes; received * bytes in responses) (glob)
  $ tail -2 errors.log
  $ hg -R clone-while-split verify
  checking changesets
  checking manifests
  crosschecking files in changesets and manifests
  checking files
  checking dirstate
  checked 3 changesets with 3 changes to 1 files
  $ hg -R clone-while-split tip
  changeset:   2:dbd9854c38a6
  tag:         tip
  user:        test
  date:        Thu Jan 01 00:00:00 1970 +0000
  summary:     c
  
  $ hg -R clone-while-split debug-revlog-index some-file
     rev linkrev       nodeid    p1-nodeid    p2-nodeid
       0       0 ed70cecbc103 000000000000 000000000000
       1       1 7241018db64c ed70cecbc103 000000000000
       2       2 fa1120531cc1 7241018db64c 000000000000
  $ hg -R server phase --rev 'all()'
  0: draft
  1: draft
  2: draft
  3: draft
  $ hg -R clone-while-split phase --rev 'all()'
  0: draft
  1: draft
  2: draft

subsequent pull work

  $ hg -R clone-while-split pull
  pulling from http://localhost:$HGPORT1/
  searching for changes
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 1 changes to 1 files
  new changesets df05c6cb1406 (1 drafts)
  (run 'hg update' to get a working copy)

  $ hg -R clone-while-split debug-revlog-index some-file
     rev linkrev       nodeid    p1-nodeid    p2-nodeid
       0       0 ed70cecbc103 000000000000 000000000000
       1       1 7241018db64c ed70cecbc103 000000000000
       2       2 fa1120531cc1 7241018db64c 000000000000
       3       3 a631378adaa3 fa1120531cc1 000000000000
  $ hg -R clone-while-split verify
  checking changesets
  checking manifests
  crosschecking files in changesets and manifests
  checking files
  checking dirstate
  checked 4 changesets with 4 changes to 1 files