Mercurial Mercurial > mercurial / changeset
summary | shortlog | changelog | graph | tags | bookmarks | branches | files | changeset | raw | help
Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
wireprotov2: add phases to "changesetdata" command
authorGregory Szorc <gregory.szorc@gmail.com>
Tue, 28 Aug 2018 18:19:23 -0700
changeset 39632 c1aacb0d76ff
parent 39631 b9e453d683a1
child 39633 ff2de4f2eb3c
wireprotov2: add phases to "changesetdata" command This commit teaches the "changesetdata" wire protocol command to emit the phase state for each changeset. This is a different approach from existing phase transfer in a few ways. Previously, if there are no new revisions (or we're not using bundle2), we perform a "listkeys" request to retrieve phase heads. And when revision data is being transferred with bundle2, phases data is encoded in a standalone bundle2 part. In both cases, phases data is logically decoupled from the changeset data and is encountered/applied after changeset revision data is received. The new wire protocol purposefully tries to more tightly associate changeset metadata (phases, bookmarks, obsolescence markers, etc) with the changeset revision and index data itself, rather than have it live as a separate entity that must be fetched and processed separately. I reckon that one reason we didn't do this before was it was difficult to add new data types/fields without breaking existing consumers. By using CBOR maps to transfer changeset data and putting clients in control of what fields are requested / present in those maps, we can easily add additional changeset data while maintaining backwards compatibility. I believe this to be a superior approach to the problem. That being said, for performance reasons, we may need to resort to alternative mechanisms for transferring data like phases. But for now, I think giving the wire protocol the ability to transfer changeset metadata next to the changeset itself is a powerful feature because it is a raw, changeset-centric data API. And if you build simple APIs for accessing the fundamental units of repository data, you enable client-side experimentation (partial clone, etc). If it turns out that we need specialized APIs or mechanisms for transferring data like phases, we can build in those APIs later. For now, I'd like to see how far we can get on simple APIs. It's worth noting that when phase data is being requested, the server will also emit changeset records for nodes in the bases specified by the "noderange" argument. This is to ensure that phase-only updates for nodes the client has are available to the client, even if no new changesets will be transferred. Differential Revision: https://phab.mercurial-scm.org/D4483
mercurial/help/internals/wireprotocolv2.txt file | annotate | diff | comparison | revisions
mercurial/wireprotov2server.py file | annotate | diff | comparison | revisions
tests/test-wireproto-command-changesetdata.t file | annotate | diff | comparison | revisions 
--- a/mercurial/help/internals/wireprotocolv2.txt	Wed Sep 12 10:01:36 2018 -0700
+++ b/mercurial/help/internals/wireprotocolv2.txt	Tue Aug 28 18:19:23 2018 -0700
@@ -113,6 +113,9 @@
    parents
       Parent revisions.
 
+   phase
+      The phase state of a revision.
+
    revision
       The raw, revision data for the changelog entry. The hash of this data
       will match the revision's node value.
@@ -126,7 +129,8 @@
 
 totalitems
    (unsigned integer) Total number of changelog revisions whose data is being
-   transferred.
+   transferred. This maps to the set of revisions in the requested node
+   range, not the total number of records that follow (see below for why).
 
 Following the map header is a series of 0 or more CBOR values. If values
 are present, the first value will always be a map describing a single changeset
@@ -144,6 +148,11 @@
    (array of bytestrings) The nodes representing the parent revisions of this
    revision. Only present if ``parents`` data is being requested.
 
+phase (optional)
+   (bytestring) The phase that a revision is in. Recognized values are
+   ``secret``, ``draft``, and ``public``. Only present if ``phase`` data
+   is being requested.
+
 revisionsize (optional)
    (unsigned integer) Indicates the size of raw revision data that follows this
    map. The following data contains a serialized form of the changeset data,
@@ -160,10 +169,19 @@
 
 Nodes from ``nodes`` are emitted before nodes from ``noderange``.
 
+The set of changeset revisions emitted may not match the exact set of
+changesets requested. Furthermore, the set of keys present on each
+map may vary. This is to facilitate emitting changeset updates as well
+as new revisions.
+
+For example, if the request wants ``phase`` and ``revision`` data,
+the response may contain entries for each changeset in the common nodes
+set with the ``phase`` key and without the ``revision`` key in order
+to reflect a phase-only update.
+
 TODO support different revision selection mechanisms (e.g. non-public, specific
 revisions)
 TODO support different hash "namespaces" for revisions (e.g. sha-1 versus other)
-TODO support emitting phases data
 TODO support emitting bookmarks data
 TODO support emitting obsolescence data
 TODO support filtering based on relevant paths (narrow clone)
--- a/mercurial/wireprotov2server.py	Wed Sep 12 10:01:36 2018 -0700
+++ b/mercurial/wireprotov2server.py	Tue Aug 28 18:19:23 2018 -0700
@@ -511,6 +511,7 @@
             # list.
 
     seen.clear()
+    publishing = repo.publishing()
 
     if outgoing:
         repo.hook('preoutgoing', throw=True, source='serve')
@@ -519,6 +520,19 @@
         b'totalitems': len(outgoing),
     }
 
+    # The phases of nodes already transferred to the client may have changed
+    # since the client last requested data. We send phase-only records
+    # for these revisions, if requested.
+    if b'phase' in fields and noderange is not None:
+        # TODO skip nodes whose phase will be reflected by a node in the
+        # outgoing set. This is purely an optimization to reduce data
+        # size.
+        for node in noderange[0]:
+            yield {
+                b'node': node,
+                b'phase': b'public' if publishing else repo[node].phasestr()
+            }
+
     # It is already topologically sorted by revision number.
     for node in outgoing:
         d = {
@@ -528,6 +542,13 @@
         if b'parents' in fields:
             d[b'parents'] = cl.parents(node)
 
+        if b'phase' in fields:
+            if publishing:
+                d[b'phase'] = b'public'
+            else:
+                ctx = repo[node]
+                d[b'phase'] = ctx.phasestr()
+
         revisiondata = None
 
         if b'revision' in fields:
--- a/tests/test-wireproto-command-changesetdata.t	Wed Sep 12 10:01:36 2018 -0700
+++ b/tests/test-wireproto-command-changesetdata.t	Tue Aug 28 18:19:23 2018 -0700
@@ -3,6 +3,10 @@
   $ hg init server
   $ enablehttpv2 server
   $ cd server
+  $ cat >> .hg/hgrc << EOF
+  > [phases]
+  > publish = false
+  > EOF
   $ echo a0 > a
   $ echo b0 > b
 
@@ -13,6 +17,7 @@
   $ hg commit -m 'commit 1'
   $ echo b2 > b
   $ hg commit -m 'commit 2'
+  $ hg phase --public -r .
 
   $ hg -q up -r 0
   $ echo a2 > a
@@ -365,6 +370,57 @@
     }
   ]
 
+Phase data is transferred upon request
+
+  $ sendhttpv2peer << EOF
+  > command changesetdata
+  >     fields eval:[b'phase']
+  >     nodes eval:[b'\x0b\xb8\xad\x89\x4a\x15\xb1\x53\x80\xb2\xa2\xa5\xb1\x83\xe2\x0f\x2a\x4b\x28\xdd']
+  > EOF
+  creating http peer for wire protocol version 2
+  sending changesetdata command
+  s>     POST /api/exp-http-v2-0001/ro/changesetdata HTTP/1.1\r\n
+  s>     Accept-Encoding: identity\r\n
+  s>     accept: application/mercurial-exp-framing-0005\r\n
+  s>     content-type: application/mercurial-exp-framing-0005\r\n
+  s>     content-length: 76\r\n
+  s>     host: $LOCALIP:$HGPORT\r\n (glob)
+  s>     user-agent: Mercurial debugwireproto\r\n
+  s>     \r\n
+  s>     D\x00\x00\x01\x00\x01\x01\x11\xa2Dargs\xa2Ffields\x81EphaseEnodes\x81T\x0b\xb8\xad\x89J\x15\xb1S\x80\xb2\xa2\xa5\xb1\x83\xe2\x0f*K(\xddDnameMchangesetdata
+  s> makefile('rb', None)
+  s>     HTTP/1.1 200 OK\r\n
+  s>     Server: testing stub value\r\n
+  s>     Date: $HTTP_DATE$\r\n
+  s>     Content-Type: application/mercurial-exp-framing-0005\r\n
+  s>     Transfer-Encoding: chunked\r\n
+  s>     \r\n
+  s>     13\r\n
+  s>     \x0b\x00\x00\x01\x00\x02\x011
+  s>     \xa1FstatusBok
+  s>     \r\n
+  received frame(size=11; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=continuation)
+  s>     3d\r\n
+  s>     5\x00\x00\x01\x00\x02\x001
+  s>     \xa1Jtotalitems\x01\xa2DnodeT\x0b\xb8\xad\x89J\x15\xb1S\x80\xb2\xa2\xa5\xb1\x83\xe2\x0f*K(\xddEphaseFpublic
+  s>     \r\n
+  received frame(size=53; request=1; stream=2; streamflags=; type=command-response; flags=continuation)
+  s>     8\r\n
+  s>     \x00\x00\x00\x01\x00\x02\x002
+  s>     \r\n
+  s>     0\r\n
+  s>     \r\n
+  received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+  response: gen[
+    {
+      b'totalitems': 1
+    },
+    {
+      b'node': b'\x0b\xb8\xad\x89J\x15\xb1S\x80\xb2\xa2\xa5\xb1\x83\xe2\x0f*K(\xdd',
+      b'phase': b'public'
+    }
+  ]
+
 Revision data is transferred upon request
 
   $ sendhttpv2peer << EOF
@@ -483,4 +539,101 @@
     b'1b74476799ec8318045db759b1b4bcc9b839d0aa\ntest\n0 0\na\n\ncommit 3'
   ]
 
+Base nodes have just their metadata (e.g. phase) transferred
+
+  $ sendhttpv2peer << EOF
+  > command changesetdata
+  >     fields eval:[b'phase', b'parents', b'revision']
+  >     noderange eval:[[b'\x33\x90\xef\x85\x00\x73\xfb\xc2\xf0\xdf\xff\x22\x44\x34\x2c\x8e\x92\x29\x01\x3a'], [b'\x0b\xb8\xad\x89\x4a\x15\xb1\x53\x80\xb2\xa2\xa5\xb1\x83\xe2\x0f\x2a\x4b\x28\xdd', b'\xea\xe5\xf8\x2c\x2e\x62\x23\x68\xd2\x7d\xae\xcb\x76\xb7\xe3\x93\xd0\xf2\x42\x11']]
+  > EOF
+  creating http peer for wire protocol version 2
+  sending changesetdata command
+  s>     POST /api/exp-http-v2-0001/ro/changesetdata HTTP/1.1\r\n
+  s>     Accept-Encoding: identity\r\n
+  s>     accept: application/mercurial-exp-framing-0005\r\n
+  s>     content-type: application/mercurial-exp-framing-0005\r\n
+  s>     content-length: 141\r\n
+  s>     host: $LOCALIP:$HGPORT\r\n (glob)
+  s>     user-agent: Mercurial debugwireproto\r\n
+  s>     \r\n
+  s>     \x85\x00\x00\x01\x00\x01\x01\x11\xa2Dargs\xa2Ffields\x83EphaseGparentsHrevisionInoderange\x82\x81T3\x90\xef\x85\x00s\xfb\xc2\xf0\xdf\xff"D4,\x8e\x92)\x01:\x82T\x0b\xb8\xad\x89J\x15\xb1S\x80\xb2\xa2\xa5\xb1\x83\xe2\x0f*K(\xddT\xea\xe5\xf8,.b#h\xd2}\xae\xcbv\xb7\xe3\x93\xd0\xf2B\x11DnameMchangesetdata
+  s> makefile('rb', None)
+  s>     HTTP/1.1 200 OK\r\n
+  s>     Server: testing stub value\r\n
+  s>     Date: $HTTP_DATE$\r\n
+  s>     Content-Type: application/mercurial-exp-framing-0005\r\n
+  s>     Transfer-Encoding: chunked\r\n
+  s>     \r\n
+  s>     13\r\n
+  s>     \x0b\x00\x00\x01\x00\x02\x011
+  s>     \xa1FstatusBok
+  s>     \r\n
+  received frame(size=11; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=continuation)
+  s>     239\r\n
+  s>     1\x02\x00\x01\x00\x02\x001
+  s>     \xa1Jtotalitems\x03\xa2DnodeT3\x90\xef\x85\x00s\xfb\xc2\xf0\xdf\xff"D4,\x8e\x92)\x01:EphaseFpublic\xa4DnodeTu\x92\x91~\x1c>\x82g|\xb0\xa4\xbcq\\\xa2]\xd1-(\xc1Gparents\x82T3\x90\xef\x85\x00s\xfb\xc2\xf0\xdf\xff"D4,\x8e\x92)\x01:T\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00EphaseFpublicLrevisionsize\x18?X?7f144aea0ba742713887b564d57e9d12f12ff382\n
+  s>     test\n
+  s>     0 0\n
+  s>     a\n
+  s>     b\n
+  s>     \n
+  s>     commit 1\xa4DnodeT\x0b\xb8\xad\x89J\x15\xb1S\x80\xb2\xa2\xa5\xb1\x83\xe2\x0f*K(\xddGparents\x82Tu\x92\x91~\x1c>\x82g|\xb0\xa4\xbcq\\\xa2]\xd1-(\xc1T\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00EphaseFpublicLrevisionsize\x18=X=37f0a2d1c28ffe4b879109a7d1bbf8f07b3c763b\n
+  s>     test\n
+  s>     0 0\n
+  s>     b\n
+  s>     \n
+  s>     commit 2\xa4DnodeT\xea\xe5\xf8,.b#h\xd2}\xae\xcbv\xb7\xe3\x93\xd0\xf2B\x11Gparents\x82T3\x90\xef\x85\x00s\xfb\xc2\xf0\xdf\xff"D4,\x8e\x92)\x01:T\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00EphaseEdraftLrevisionsize\x18=X=1b74476799ec8318045db759b1b4bcc9b839d0aa\n
+  s>     test\n
+  s>     0 0\n
+  s>     a\n
+  s>     \n
+  s>     commit 3
+  s>     \r\n
+  received frame(size=561; request=1; stream=2; streamflags=; type=command-response; flags=continuation)
+  s>     8\r\n
+  s>     \x00\x00\x00\x01\x00\x02\x002
+  s>     \r\n
+  s>     0\r\n
+  s>     \r\n
+  received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+  response: gen[
+    {
+      b'totalitems': 3
+    },
+    {
+      b'node': b'3\x90\xef\x85\x00s\xfb\xc2\xf0\xdf\xff"D4,\x8e\x92)\x01:',
+      b'phase': b'public'
+    },
+    {
+      b'node': b'u\x92\x91~\x1c>\x82g|\xb0\xa4\xbcq\\\xa2]\xd1-(\xc1',
+      b'parents': [
+        b'3\x90\xef\x85\x00s\xfb\xc2\xf0\xdf\xff"D4,\x8e\x92)\x01:',
+        b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+      ],
+      b'phase': b'public',
+      b'revisionsize': 63
+    },
+    b'7f144aea0ba742713887b564d57e9d12f12ff382\ntest\n0 0\na\nb\n\ncommit 1',
+    {
+      b'node': b'\x0b\xb8\xad\x89J\x15\xb1S\x80\xb2\xa2\xa5\xb1\x83\xe2\x0f*K(\xdd',
+      b'parents': [
+        b'u\x92\x91~\x1c>\x82g|\xb0\xa4\xbcq\\\xa2]\xd1-(\xc1',
+        b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+      ],
+      b'phase': b'public',
+      b'revisionsize': 61
+    },
+    b'37f0a2d1c28ffe4b879109a7d1bbf8f07b3c763b\ntest\n0 0\nb\n\ncommit 2',
+    {
+      b'node': b'\xea\xe5\xf8,.b#h\xd2}\xae\xcbv\xb7\xe3\x93\xd0\xf2B\x11',
+      b'parents': [
+        b'3\x90\xef\x85\x00s\xfb\xc2\xf0\xdf\xff"D4,\x8e\x92)\x01:',
+        b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
+      ],
+      b'phase': b'draft',
+      b'revisionsize': 61
+    },
+    b'1b74476799ec8318045db759b1b4bcc9b839d0aa\ntest\n0 0\na\n\ncommit 3'
+  ]
+
   $ cat error.log
mercurial