Martin von Zweigbergk <martinvonz@google.com> [Thu, 27 Sep 2018 09:56:13 -0700] rev 39906
context: stop catching and re-raising FilteredRepoLookupError
FilteredRepoLookupError is only raised by changectx's constructor and
the higher-level scmutil.revsymbol(), so there's no need to catch it
in changectx's constructor.
Differential Revision: https://phab.mercurial-scm.org/D4806
Matt Harbison <matt_harbison@yahoo.com> [Fri, 28 Sep 2018 22:18:45 -0400] rev 39905
util: use a context manager in readlock()
Matt Harbison <matt_harbison@yahoo.com> [Tue, 25 Sep 2018 21:16:12 -0400] rev 39904
py3: convert os.readlink() path to native strings on Windows
Windows insisted that it needs to be str. I skipped the stuff in the posix
module, and left `tests/f` and `run-tests.py` alone for now.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 29 Sep 2018 02:02:35 -0400] rev 39903
py3: suppress the output from .write() calls in the remaining tests
Matt Harbison <matt_harbison@yahoo.com> [Fri, 28 Sep 2018 23:17:06 -0400] rev 39902
py3: conditionalize json float precision difference in test-debugcommands.t
Matt Harbison <matt_harbison@yahoo.com> [Fri, 28 Sep 2018 22:39:18 -0400] rev 39901
py3: byteify extension in test-debugcommands.t
Boris Feld <boris.feld@octobus.net> [Thu, 27 Sep 2018 16:55:06 +0200] rev 39900
pullreport: rev duplicated and extinct into account
If we already have some obsolete and hidden nodes locally and the server send
them again to you, it seems useful to point it out instead of being silent about
it.
Boris Feld <boris.feld@octobus.net> [Thu, 27 Sep 2018 16:52:25 +0200] rev 39899
pullreport: issue a message about "extinct" pulled changesets
Changeset pulled from a remote repository while already obsolete locally can
end up hidden after the pull. Hiding obsolete changesets is a good behavior
but silently "skipping" some of the pulled content can get confusing.
We now detect this situation and emit a message about it. The message is
simple and the wording could be improved, however, we focus on the detection
here. Evolution is still an experimental feature, so the output is open to
changes.
In particular, we could point out at the latest successors of the obsolete
changesets, however, it can get tricky is there are many of them. So we delay
these improvements to another adventure.
Another easy improvement would be to merge this message with the previous line
about the new nodes and their phases.
This is a good example of cases where we can only transmit a limited amount of
data to users by default. We need some sort of "transaction journal" we could
point the user to.
Boris Feld <boris.feld@octobus.net> [Thu, 27 Sep 2018 17:00:00 +0200] rev 39898
pullreport: skip or rework some early return
We are about to add more logic in this report. Before that, we need it to not
quit so early.
Boris Feld <boris.feld@octobus.net> [Thu, 27 Sep 2018 16:35:10 +0200] rev 39897
pullreport: skip filtered revs instead of obsolete ones
Obsolescence is closely related to visibility but still a distinct concept. We
can receive changesets that are obsolete but visible (eg: when pulling
orphans). Such changeset should be reported too. In addition, the filtering
level can be anything, we should respect it.
Boris Feld <boris.feld@octobus.net> [Thu, 27 Sep 2018 18:06:13 +0200] rev 39896
pullreport: add a test to show misreporting of visible changeset
The current code ignores all obsolete changesets including the visible one. We
add a test showing this behavior before fixing the behavior.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 26 Sep 2018 10:38:37 -0700] rev 39895
repo: don't look up context for tip node if it's not needed
We were doing repo['tip'].node() or similar in a few places where
repo.changelog.tip() would be enough.
Differential Revision: https://phab.mercurial-scm.org/D4781
Martin von Zweigbergk <martinvonz@google.com> [Wed, 26 Sep 2018 22:17:34 -0700] rev 39894
repo: look up nullrev context by revnum, not symbolic name
I think lookup of the 'null' symbol should be done via
scmutil.revsymbol() and repo['null'] is only supported for historical
reasons. However, repo[nullrev] is fine, so we can switch to that
instead of switching to scmutil.revsymbol('null').
Differential Revision: https://phab.mercurial-scm.org/D4780
Martin von Zweigbergk <martinvonz@google.com> [Wed, 26 Sep 2018 22:17:26 -0700] rev 39893
bundle: consistently put revnums in "base" collection
The "base" collection contains revnums, except that it can also
contain the "null" symbol. That's a little weird. Let's be consistent.
Differential Revision: https://phab.mercurial-scm.org/D4779
Matt Harbison <matt_harbison@yahoo.com> [Thu, 27 Sep 2018 22:07:12 -0400] rev 39892
py3: ensure printing to stdout uses str in test-hgweb-no-request-uri.t
Boris Feld <boris.feld@octobus.net> [Thu, 27 Sep 2018 13:57:50 -0700] rev 39891
scmutil: accept multiple predecessors in 'replacements' (API)
This changeset makes 'cleanupnodes' accepts multiple predecessors as
`replacements` keys. The same as it accepts multiple successors as
`replacements` values. To avoid breaking all callers, the old and new ways are
currently valid at the same time. We'll deprecate and drop the old way later.
This change is the first step toward a better tracking of "fold" event in the
evolution history. While working on the "rewind" command (in the evolve
extension), we realized that first class tracking of folds are necessary.
We already have good tracking of splits. When walking the evolution history
from predecessors to successors, that makes for a clear distinction between
having multiple successors because of the actual splitting of a changeset or
content-divergences.
The "rewind" command allows restoring older evolution of a stack of
changesets. One of its mode walks the evolution history to automatically find
appropriate predecessors. This means walking from successors to predecessors.
In this case, we need to be able to make the same distinction between an
actual fold and other cases. So we will have to track folds explicitly.
This changesets only focus on making it possible to express fold at the
`cleanupnodes` API level. The actual tracking will be implemented later.
Boris Feld <boris.feld@octobus.net> [Thu, 27 Sep 2018 13:54:37 -0700] rev 39890
scmutil: expand long "one-liner"
When a one-liner gets 3 lines longs, it loses its expressivity benefits. We
expand it into a simple for loop. This makes future changes of the code in
that area clearer.
Boris Feld <boris.feld@octobus.net> [Thu, 20 Sep 2018 17:47:05 +0200] rev 39889
shelve: find shelvedctx from bundle even if they are already in the repo
We use the new "duplicates" node tracking to find the tip of the bundle even if
it already exists in the repository.
Such logic is not supposed to be needed in theory. If the shelve was made using
internal-phase, we already know its node. Otherwise, the bundle content should
have been stripped. However, handling it makes the shelve code more robust and
provide a good example of "revduplicates" usage.
Boris Feld <boris.feld@octobus.net> [Wed, 19 Sep 2018 12:19:28 +0200] rev 39888
shelve: return the shelved node as part of bundle application
It make sense to have the function in charge of unbundling the shelved revision
also return the node of that revision (when the data is in the bundle).
This will help us to handle unnatural state where the unshelved change already
exists in the repository.
Boris Feld <boris.feld@octobus.net> [Thu, 20 Sep 2018 11:18:28 +0200] rev 39887
changelog: keep track of duplicated node in the transaction adding them
The transaction is already tracking the new nodes. We now tracks the
"duplicates" in the same location.
Boris Feld <boris.feld@octobus.net> [Wed, 19 Sep 2018 21:02:47 +0200] rev 39886
revlog: add a callback "tracking" duplicate node addition
If a changegroup contains node already added to the repository, they will be
skipped. Skipping them is the right behavior (we don't need to store things
twice), but it can hide some information to the code doing the unbundle (eg:
shelve looking for the tip of the bundle).
The first step to improve this situation is to add a low level callback. We do
not need this tracking on all revlog, so actual tracking will be added in the
next changeset.
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Wed, 26 Sep 2018 18:30:19 -0400] rev 39885
logtoprocess: define $HG for children processes
So they can compute the hg version for instance.
Differential Revision: https://phab.mercurial-scm.org/D4768
Matt Harbison <matt_harbison@yahoo.com> [Wed, 26 Sep 2018 22:21:25 -0400] rev 39884
py3: mask out None type when printing in `debuglocks`
Apparently, %b doesn't allow None.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 26 Sep 2018 21:25:18 -0400] rev 39883
py3: ensure standard exceptions use `str` type strings in windows.py
See also edaa40dc5fe5.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 26 Sep 2018 20:49:28 -0400] rev 39882
py3: replace a StandardError reference
This doesn't exist on py3, and the standard way of handling this seems to be to
catch both exceptions.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 15:19:52 -0700] rev 39881
storageutil: extract revision number iteration
This code is a bit quirky (and possibly buggy). It will likely be used
by multiple storage backends. Let's extract it so it is reusable.
Differential Revision: https://phab.mercurial-scm.org/D4757
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 14:54:28 -0700] rev 39880
storageutil: new function for extracting metadata-less content from text
Other storage backends will want to do this.
I'm not concerned about Python function call overhead because I
expect self.revision() to dwarf the function call overhead time,
since self.revision() requires multiple function calls and may
involve decompression in the common case.
Differential Revision: https://phab.mercurial-scm.org/D4756
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 14:33:45 -0700] rev 39879
storageutil: move _censoredtext() from revlog
This seems like generic functionality we'll want to use from
non-revlog storage backends.
Differential Revision: https://phab.mercurial-scm.org/D4755
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 14:31:31 -0700] rev 39878
storageutil: move metadata parsing and packing from revlog (API)
Parsing and writing of revision text metadata is likely identical
across storage backends. Let's move the code out of revlog so we
don't need to import the revlog module in order to use it.
Differential Revision: https://phab.mercurial-scm.org/D4754
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 14:23:54 -0700] rev 39877
storageutil: new module for storage primitives (API)
There will exist common code between storage backends. It would
be nice to have a central place to put that code.
This commit attempts to create that place by creating the
"storageutil" module.
The first thing we move is revlog.hash(), which is the function for
computing the SHA-1 hash of revision fulltext and parents.
Differential Revision: https://phab.mercurial-scm.org/D4753
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 13:35:50 -0700] rev 39876
filelog: stop proxying deltaparent() (API)
deltaparent() obtains the revision number of the base revision a
delta in storage is stored against. It is highly revlog-centric and
may not apply to other storage backends. As a result, it doesn't
belong on the generic file storage interface.
This method/proxy is no longer used in core. The last consumer was
probably changegroup code and went away with the transition to
emitrevisions().
Differential Revision: https://phab.mercurial-scm.org/D4751
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 12:49:17 -0700] rev 39875
filelog: stop proxying rawsize() (API)
This method is no longer used by external consumers. The API is
quite low-level and is effectively len(revision(raw=True)). I don't
see a compelling reason to keep it around.
Let's drop the API and make the file storage interface simpler.
Differential Revision: https://phab.mercurial-scm.org/D4750
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 12:42:03 -0700] rev 39874
filelog: stop proxying "opener" (API)
The last consumer of it in upgrade code was removed as part of the
previous commit. This attribute is revlog specific (because it
assumes the existence of a vfs for performing I/O on tracked file
data) and therefore isn't appropriate for a generic storage interface.
So nuke it.
Differential Revision: https://phab.mercurial-scm.org/D4749
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 11:16:33 -0700] rev 39873
filelog: stop proxying flags() (API)
Per-revision storage flags are kinda a revlog-centric API. (Except for
the fact that changegroup uses the same integer flags as revlog does
and there's minimal verification that the server's flags map to the
client's storage flags - but that's another problem.)
The last user of flags() was in verify.py and that code was just moved
into revlog.py and is accessed behind the verifyintegrity() file
storage API.
Since there are no more consumers, let's drop the proxy and remove
the method from the file storage interface.
This commit only drops the dedicated API for reading a single
revision's storage flags: we still support reading and writing flags
through the bulk data retrieval and add revision APIs. And since
changegroups encode revlog integer flags over the wire, we'll always
need to support flags at some level. The removal of individual storage
flags may be too premature. But since flags() is now unused, I'd like
to see how far we can get without that dedicated API - especially
since it uses revision numbers instead of nodes.
Differential Revision: https://phab.mercurial-scm.org/D4746
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 11:27:47 -0700] rev 39872
revlog: move revision verification out of verify
File revision verification is performing low-level checks of file
storage, namely that flags are appropriate and revision data can
be resolved.
Since these checks are somewhat revlog-specific and may not
be appropriate for alternate storage backends, this commit moves
those checks from verify.py to revlog.py.
Because we're now emitting warnings/errors that apply to specific
revisions, we taught the iverifyproblem interface to expose the
problematic node and to report this node in verify output. This
was necessary to prevent unwanted test changes.
After this change, revlog.verifyintegrity() and file verify code
in verify.py both iterate over revisions and resolve their fulltext.
But they do so in separate loops. (verify.py needs to resolve
fulltexts as part of calling renamed() - at least when using revlogs.)
This should add overhead.
But on the mozilla-unified repo:
$ hg verify
before: time: real 700.640 secs (user 585.520+0.000 sys 23.480+0.000)
after: time: real 682.380 secs (user 570.370+0.000 sys 22.240+0.000)
I'm not sure what's going on. Maybe avoiding the filelog attribute
proxies shaved off enough time to offset the losses? Maybe fulltext
resolution has less overhead than I thought?
I've left a comment indicating the potential for optimization. But
because it doesn't produce a performance regression on a large
repository, I'm not going to worry about it.
Differential Revision: https://phab.mercurial-scm.org/D4745
Martin von Zweigbergk <martinvonz@google.com> [Wed, 26 Sep 2018 12:06:44 -0700] rev 39871
tests: de-flake test-narrow-debugrebuilddirstate.t
If the dirstate gets written much later (usually 1-2 s, depending on
FS) than the working copy file (there's only one), then the `hg
debugdirstate` command will include a timestamp. There's nothing wrong
with that, so we should just allow it.
Differential Revision: https://phab.mercurial-scm.org/D4758
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 12:39:34 -0700] rev 39870
upgrade: use storageinfo() for obtaining storage metadata
Let's switch to our new API for obtaining information about storage.
This eliminates the last consumer of rawsize() and the opener proxy
from the file storage interface!
Differential Revision: https://phab.mercurial-scm.org/D4748
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 11:56:48 -0700] rev 39869
revlog: add method for obtaining storage info (API)
We currently have a handful of methods on the file and manifest
storage interfaces for obtaining metadata about storage. e.g.
files() is used to obtain the files backing storage. rawsize()
is to quickly compute the size of tracked revisions without resolving
their fulltext.
Code in upgrade and stream clone make heavy use of these methods.
The existing APIs are generic and don't necessarily have the
specialization that we need going forward. For example, files()
doesn't distinguish between exclusive storage and shared storage.
This makes stream clone difficult to implement when e.g. there may
be a single file backing storage for multiple tracked paths. It
also makes reporting difficult, as we don't know how many bytes are
actually used by storage since we can't easily identify shared files.
This commit implements a new method for obtaining storage metadata.
It is designed to accept arguments specifying what metadata to request
and to return a dict with those fields populated. We /could/ make
each of these attributes a separate method. But this is a specialized
API and I'm trying to avoid method bloat on the interfaces. There is
also the possibility that certain callers will want to obtain multiple
fields in different combinations and some backends may have performance
issues obtaining all that data via separate method calls.
Simple storage integration tests have been added. For now, we assume
fields can't be "None" (ignoring the interface documentation). We can
revisit this later.
Differential Revision: https://phab.mercurial-scm.org/D4747
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 11:27:41 -0700] rev 39868
lfs: drop unused import
A recent change dropped the last user of this module.
Differential Revision: https://phab.mercurial-scm.org/D4744
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 10:08:58 -0700] rev 39867
filelog: drop _generaldelta attribute (API)
With changegroup moving to emitrevisions(), this revlog-specific
attribute is no longer used and can be deleted. Good riddance.
Differential Revision: https://phab.mercurial-scm.org/D4727
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 09:59:19 -0700] rev 39866
revlog: drop emitrevisiondeltas() and associated functionality (API)
emitrevisions() is the future!
Differential Revision: https://phab.mercurial-scm.org/D4726
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 21 Sep 2018 18:47:04 -0700] rev 39865
changegroup: port to emitrevisions() (issue5976)
We now have a unified API for emitting revision data from a storage
backend. It handles sorting nodes and the complicated delta versus
revision decisions for us.
This commit ports changegroup to that API.
There should be no behavior changes for changegroups not using
ellipsis. And lack of test changes seems to confirm that.
There are some changes for ellipsis mode, however.
Before, when sending an ellipsis revision, we would always send a
fulltext revision (as opposed to a delta). There was a TODO tracking
this open item.
One of the things the emitrevisions() API does for us is figure out
whether we can safely emit a delta. So, it is now possible for
ellipsis revisions to be sent as deltas! (It does this by not
assuming parent/ancestor revisions are available and tracking which
revisions have been sent out.)
Because we eliminated the list of revision delta request objects,
performance has improved substantially:
$ hg perfchangegroupchangelog
before: ! wall 24.348077 comb 24.330000 user 24.140000 sys 0.190000 (best of 3)
after: ! wall 18.245911 comb 18.240000 user 18.100000 sys 0.140000 (best of 3)
That's a lot of overhead for creating a few hundred thousand Python
objects!
This is still a little slower than 4.7. Probably due to 23d582ca
introducing a type for the revision/delta results. There is
potentially room to optimize. But at some point we need to abstract
storage in order to support alternate storage backends. Unfortunately
that means using a Python data structure to represent results. And
unfortunately there is overhead with every new Python object created.
Differential Revision: https://phab.mercurial-scm.org/D4725
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 09:48:02 -0700] rev 39864
wireprotov2server: port to emitrevisions()
We now have a proper storage API to request data on multiple
revisions. We can drop it into wire protocol version 2 with
minimal effort.
The new API handles pretty much everything we were doing manually to
build up the delta request. So we were able to delete a lot of code.
As a bonus, wireprotov2 code is no longer accessing some low-level
storage APIs. This includes the assumption that a node has an
associated numeric revision number! This should make it drastically
simpler to implement a server that doesn't have the concept of
revision numbers.
Differential Revision: https://phab.mercurial-scm.org/D4724
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 21 Sep 2018 14:54:59 -0700] rev 39863
tests: use more complex file storage test
The previous test was attempting to to test delta storage behavior.
It didn't do a very good job at it because there was a good chance
a delta wasn't being used in storage.
Let's switch the test to yield a delta in storage so an upcoming
change to delegate delta logic to storage has the desired effect.
Differential Revision: https://phab.mercurial-scm.org/D4723
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 21 Sep 2018 14:28:21 -0700] rev 39862
revlog: new API to emit revision data
I recently refactored changegroup generation code to make it more
storage agnostic. I made significant progress. But there is still
a bit of work to be done. Specifically:
* Changegroup code is looking at low-level storage attributes to
influence sorting. Sorting should be done at the storage layer.
* The linknode lookup and sorting code for ellipsis is very
complicated.
* Linknodes are just generally wonky because e.g. file storage doesn't
know how to translate a linkrev to a changelog node.
* We regressed performance when introducing the request-response
objects.
Having thought about this problem a bit, I think I've come up with
a better interface for emitting revision deltas.
This commit defines and implements that interface. See the docstring
in repository.py for more info.
This API adds 3 notable features over the previous one.
First, it defers node ordering to the storage implementation in
the common case but allows overriding as necessary. We have a
facility for requesting an exact ordering (used in ellipsis
mode). We have another facility for storage order (used for changelog).
Second, we have an argument specifying assumptions about parents
revisions. This can be used to force a fulltext revision when we
don't know the receiver has a parent revision to delta against.
Third, we can control whether revision data is emitted. This makes
the API suitable as a generic "index data retrieval" API as well
as for producing revision deltas - possibly in the same operation!
The new API is much simpler: we no longer need a complicated "request"
object to encapsulate the delta generation request. I'm optimistic
this will restore performance loss associated with
emitrevisiondeltas().
Storage unit tests for the new API have been implemented.
Future commits will port existing consumers of emitrevisiondeltas()
to the new API then remove emitrevisiondeltas().
Differential Revision: https://phab.mercurial-scm.org/D4722
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 09:41:42 -0700] rev 39861
changegroup: remove reordering control (BC)
This logic - including the experimental bundle.reorder option -
was originally added in a8e3931e3fb5 in 2011 and then later ported
to changegroup.py.
The intent of this option and associated logic is to control
the ordering of revisions in deltagroups in changegroups. At the
time it was implemented, only changegroup version 1 existed
and generaldelta revlogs were just coming into the world. Changegroup
version 1 requires that deltas be made against the last revision
sent over the wire. Used with generaldelta, this created an
impedance mismatch of sorts and resulted in changegroup producers
spending a lot of time recomputing deltas.
Revision reordering was introduced so outgoing revisions would be
sent in "generaldelta order" and producers would be able to
reuse internal deltas from storage.
Later on, we introduced changegroup version 2. It supported denoting
which revision a delta was against. So we no longer needed to
sort outgoing revisions to ensure optimal delta generation from the
producer. So, subsequent changegroup versions disabled reordering.
We also later made the changelog not store deltas by default. And
we also made the changelog send out deltas in storage order. Why we
do this for changelog, I'm not sure. Maybe we want to preserve revision
order across clones? It doesn't really matter for this commit.
Fast forward to 2018. We want to abstract storage backends. And having
changegroup code require knowledge about how deltas are stored
internally interferes with that goal.
This commit removes reordering control from changegroup generation.
After this commit, the reordering behavior is:
* The changelog is always sent out in storage order (no behavior
change).
* Non-changelog generaldelta revlogs are reordered to always be in DAG
topological order (previously, generaldelta revlogs would be emitted
in storage order for version 2 and 3 changegroups).
* Non-changelog non-generaldelta revlogs are sent in storage order (no
behavior change).
* There exists no config option to override behavior.
The big difference here is that generaldelta revlogs now *always* have
their revisions sorted in DAG order before going out over the wire. This
behavior was previously only done for changegroup version 1. Version 2
and version 3 changegroups disabled reordering because the interchange
format supported encoding arbitrary delta parents, so reordering wasn't
strictly necessary.
I can think of a few significant implications for this change.
Because changegroup receivers will now see non-changelog revisions
in DAG order instead of storage order, the internal storage order of
manifests and files may differ substantially between producer and
consumer. I don't think this matters that much, since the storage
order of manifests and files is largely hidden from users. Only
the storage order of changelog matters (because `hg log` shows the
changelog in storage order). I don't think there should be any
controversy here.
The reordering of revisions has implications for changegroup producers.
Previously, generaldelta revlogs would be emitted in storage order.
And in the common case, the internally-stored delta could effectively
be copied from disk into the deltagroup delta. This meant that emitting
delta groups for generaldelta revlogs would be mostly linear read I/O.
This is desirable for performance. With us now reordering generaldelta
revlog revisions in DAG order, the read operations may use more random
I/O instead of sequential I/O. This could result in performance
loss. But with the prevalence of SSDs and fast random I/O, I'm not
too worried. (Note: the optimal emission order for revlogs is actually
delta encoding order. But the changegroup code wasn't doing that before
or after this change. We could potentially implement that in a later
commit.)
Changegroups in DAG order will have implications for receivers.
Previously, receiving storage order might mean seeing a number of
interleaved branches. This would mean long delta chains, sparse
I/O, and possibly more fulltext revisions instead of deltas, blowing
up storage storage. (This is the same set of problems that sparse
revlogs aims to address.) With the producer now sending revisions in DAG
order, the receiver also stores revisions in DAG order. That means
revisions for the same DAG branch are all grouped together. And this
should yield better storage outcomes. In other words, sending the
reordered changegroup allows the receiver to have better storage
order and for the producer to not propagate its (possibly sub-optimal)
internal storage order.
On the mozilla-unified repository, this change influences bundle
generation:
$ hg bundle -t none-v2 -a
before: time: real 355.680 secs (user 256.790+0.000 sys 16.820+0.000)
after: time: real 382.950 secs (user 281.700+0.000 sys 17.690+0.000)
before: 7,150,228,967 bytes (uncompressed)
after: 7,041,556,273 bytes (uncompressed)
before: 1,669,063,234 bytes (zstd l=3)
after: 1,628,598,830 bytes (zstd l=3)
$ hg unbundle
before: time: real 511.910 secs (user 466.750+0.000 sys 32.680+0.000)
after: time: real 487.790 secs (user 443.940+0.000 sys 30.840+0.000)
00manifest.d size:
source: 274,924,292 bytes
before: 304,741,626 bytes
after: 245,252,087 bytes
.hg/store total file size:
source: 2,649,133,490
before: 2,680,888,130
after: 2,627,875,673
We see the bundle size drop. That's probably because if a revlog
internally isn't storing a delta, it will choose to delta against
the last emitted revision. And on repos with interleaved branches
(like mozilla-unified), the previous revision could be an unrelated
branch and therefore be a large delta. But with this patch, the
previous revision is likely p1 or p2 and a delta should be small.
We also see the manifest size drop by ~50 MB. It's worth noting that
the manifest actually *increased* in size by ~25 MB in the old
strategy and decreased ~25 MB from its source in the new strategy.
Again, my explanation for this is that the DAG ordering in the
changegroup is resulting in better grouping of revisions in the
receiver, which results in more compact delta chains and higher
storage efficiency.
Unbundle time also dropped. I suspect this is due to the revlog having
to work less to compute deltas since the incoming deltas are more
optimal. i.e. the receiver spends less time resolving fulltext
revisions as incoming deltas bounce around between DAG branches and
delta chains.
We also see bundle generation time increase. This is not desirable.
However, the regression is only significant on the original repository:
if we generate a bundle from the repository created from the new,
always reordered bundles, we're close to baseline (if not at it with
expected noise):
$ hg bundle -t none-v2 -a
before (original): time: real 355.680 secs (user 256.790+0.000 sys 16.820+0.000)
after (original): time: real 382.950 secs (user 281.700+0.000 sys 17.690+0.000)
after (new repo): time: real 362.280 secs (user 260.300+0.000 sys 17.700+0.000)
This regression is a bit worrying because it will impact serving
canonical repositories (that don't have optimal internal storage
unless they are reordered - possibly as part of running
`hg debugupgraderepo`). However, this regression will only be
noticed by very large changegroups. And I'm guessing/hoping that
any repository that large is using clonebundles to mitigate server
load.
Again, sending DAG order isn't the optimal send order for servers:
sending in storage-delta order is. But in order to enable
storage-optimal send order, we'll need a storage API that handles
sorting. Future commits will introduce such an API.
Differential Revision: https://phab.mercurial-scm.org/D4721
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 20 Sep 2018 19:31:07 -0700] rev 39860
filelog: drop index attribute (API)
The previous commit removed the last consumer of the "index" attribute
on the file storage interface. The index is an extremely low-level
data structure that is revlog specific and isn't appropriate to
expose as part of a generic storage API. There may be a market for
an efficient data structure to obtain metadata on every revision for
a file. But if there is, it should be designed using e.g. named
attributes for lookup instead of a list-like of 8-tuples.
Let's drop the attribute from filelog and remove the attribute from
the file storage interface.
Differential Revision: https://phab.mercurial-scm.org/D4720
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 09:38:27 -0700] rev 39859
upgrade: use rawsize() instead of revlog index
The revlog index is a very low-level data structure and it shouldn't
be exposed to the storage interface - at least not in its current
form.
upgrade.py is the only consumer of the index attribute on file storage
in the repository.
This commit rewrites that final consumer to use rawsize() instead of
going through the index. This is actually the more proper API to use,
as rawsize() will accurately report the size of revisions which have
a negative size in the index.
Differential Revision: https://phab.mercurial-scm.org/D4719
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 20 Sep 2018 19:20:01 -0700] rev 39858
manifest: add rawsize() proxy (API)
I'm not keen about doing this. But it unblocks efforts to remove
"index" from the file storage interface. We will probably remove this
once we have a better upgrade API in place.
Differential Revision: https://phab.mercurial-scm.org/D4718
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 09:37:19 -0700] rev 39857
upgrade: report size of backing files, not internal storage size
upgrade.py is the only consumer of filelog.index, which I'd like
to eliminate from the file storage interface.
This commit changes the upgrade code to report the storage size
of files by looking at the size of the files backing its storage
instead of looking at the index.
I'm not convinced the approach in this patch will live very long
because it is relying on low-level attributes like "opener" and
"files," which may behave very differently on non-revlog storage.
But the data is only used for reporting purposes and it does get
us one step closer to eliminating "index."
A side-effect of this change is we now report the size of the revlog
index data - not just the revision data. I think this is more
accurate.
Differential Revision: https://phab.mercurial-scm.org/D4717
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 20 Sep 2018 18:07:42 -0700] rev 39856
filelog: store filename directly on revlog instance
This attribute is only used by LFS. It is used by one of the revlog
flag processor functions, which gets an instance of the revlog - not
the file storage type. So, it makes sense to store this attribute on
the revlog instead of the filelog.
With this change, I'm pretty confident that LFS is no longer directly
accessing file storage interface members that are revlog centric. i.e.
it gets us one step closer to eliminating revlog-centric APIs from the
file storage interface!
Differential Revision: https://phab.mercurial-scm.org/D4715
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 20 Sep 2018 17:47:34 -0700] rev 39855
lfs: access revlog directly
LFS is monkeypatching filelog.filelog and is then accessing
various filelog attributes in the monkeypatched function. This is all
fine.
But some of the attributes being accessed by LFS are revlog centric
and shouldn't be exposed on the file storage interface.
This commit changes the monkeypatched functions to access proxied
attributes on self._revlog instead of self.
This should be safe to do because non-revlog repositories should not
be using filelog instances: instead they should have a separate class
to represent file storage. So it is reasonable for LFS to assume the
_revlog attribute exists and points to a revlog.
Differential Revision: https://phab.mercurial-scm.org/D4714
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 20 Sep 2018 15:30:00 -0700] rev 39854
largefiles: automatically load largefiles extension when required (BC)
This is very similar to what we just did for LFS but for largefiles.
See recent commit messages for the rationale here.
Differential Revision: https://phab.mercurial-scm.org/D4713
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 20 Sep 2018 15:18:13 -0700] rev 39853
lfs: don't add extension to hgrc after clone or share (BC)
Now that repository loading in core supports automatically loading
the lfs extension when the "lfs" requirement is present, we no
longer need to update the .hg/hgrc of newly-created repos to load
the lfs extension!
I'm marking this as BC because it is a change in behavior. But users
should not notice unless they create an LFS repo with new Mercurial
and then attempt to use it with an old versions that doesn't support
automatic extension loading.
Differential Revision: https://phab.mercurial-scm.org/D4712
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 20 Sep 2018 15:06:43 -0700] rev 39852
localrepo: automatically load lfs extension when required (BC)
If an unrecognized requirement is present (possibly due to an unloaded
extension), the user will get an error message telling them to go to
https://mercurial-scm.org/wiki/MissingRequirement for more info.
And some requirements clearly map to known extensions shipped by
Mercurial.
This commit teaches repository loading to automatically map
requirements to extensions. We implement support for loading the
lfs extension when the "lfs" requirement is present.
This behavior feels more user-friendly to me and I'm having trouble
coming up with a compelling reason to not do it. The strongest
argument I have against is that - strictly speaking - requirements
are general repository features and there could be N providers of that
feature. e.g. in the case of LFS, there could be another extension
implementing LFS support. And the user would want to use this
non-official extension rather than the built-in one. The way this
patch implements things, the non-official extension could be
missing and Mercurial would load the official lfs extension, leading
to unexpected behavior. But this feels like a highly marginal use
case to me and doesn't outweigh the user benefit of "it just works."
If someone really wanted to e.g. use a custom LFS extension, they
could prevent the built-in one from being loaded by either defining
"extensions.lfs=/path/to/custom/extension" or "extensions.lfs=!",
as the automatic extension loading only occurs if there is no config
entry for that extension.
Differential Revision: https://phab.mercurial-scm.org/D4711
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 19 Sep 2018 13:48:59 -0700] rev 39851
lfs: add repository feature denoting the use of LFS
Whether LFS is enabled seems like a useful feature to expose. This
will also facilitate some future work around LFS feature compatibility.
Differential Revision: https://phab.mercurial-scm.org/D4710
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 19 Sep 2018 14:36:57 -0700] rev 39850
localrepo: define "features" on repository instances (API)
There are a handful of attributes/methods on repository instances that
describe the behavior of the repository. Furthermore, there is
an unbound set of repository descriptors that we may wish to expose.
For example, an extension may wish to add a descriptor and have
monkeypatched functions look for the presence of an attribute before
taking actions.
This commit introduces a "features" mechanism to allow repositories
to self-advertise an arbitrary set of strings that describe repository
behavior or capabilities.
We implement basic support for advertising a few features to give an
idea of what I want to use this for.
Differential Revision: https://phab.mercurial-scm.org/D4709
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 19 Sep 2018 17:27:37 -0700] rev 39849
localrepo: support writing shared file (API)
Now that we can create a shared repository via creation options, we
can handle other special actions related to share at repo creation time
as well.
One of the things we do after creating a shared repository is write
out a .hg/shared file containing the list of additional things to
share. Of which only "bookmarks" is supported.
We add a creation option to hold the set of additional items to
share. If items are defined, we write out the .hg/shared file at
repo creation time.
As part of this, we no longer hold the repo lock when writing the
file. I'm pretty sure we don't care about the tiny race condition
window. I'm also pretty sure the reason we used the lock was because
the vfs auditor on the repo instance complained otherwise. Since the
repo creation code doesn't have an audited vfs, we don't need to
appease it.
Because we no longer need to tell the post share hook what items
are shared, the "bookmarks" argument to that function has been
dropped, incurring an API change.
Differential Revision: https://phab.mercurial-scm.org/D4708
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 19 Sep 2018 17:05:59 -0700] rev 39848
localrepo: support shared repo creation
Previously, hg.share() had its own logic for creating a new
repository on the filesystem.
With the recent introduction of the createopts dict for passing
options to influence repository creation, it is now possible
to consolidate the repo creation code for both the normal and
shared use cases.
This commit teaches the repo creation code in localrepo to
recognize when we're creating a shared repo and to act
appropriately.
Meaningful behavior should be identical. However, there are a
few subtle changes:
* The .hg/requires file is written out in sorted order (rather
than having share-related requirements appended at end).
* The .hg directory is created with notindexed=True when a shared
repo is being created.
Differential Revision: https://phab.mercurial-scm.org/D4707
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 19 Sep 2018 16:51:57 -0700] rev 39847
localrepo: validate directories before creating any
There is no meaningful change in behavior because wdir would
already exist in the case where we raised RepoError. But I think
the code is easier to read if we do all validation first then
take actions with side-effects.
Differential Revision: https://phab.mercurial-scm.org/D4706
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 19 Sep 2018 16:15:22 -0700] rev 39846
localrepo: add missing join()
Differential Revision: https://phab.mercurial-scm.org/D4705
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 19 Sep 2018 11:38:05 -0700] rev 39845
revlog: use proper version comparison during verify
Verify appears to want to compare the changelog's revlog version
number with the version number of filelogs and error if they are
different. But what it was actually doing was comparing the full
32-bit header integer, which contains 2 shorts: 1 for the revlog
version number and 1 for feature flags.
This commit tweaks the verification code so it only looks at the
version number component of the header and emits a warning if they
differ.
The new code is more robust because it accounts for future revlog
version numbers without them needing to be special cased.
Differential Revision: https://phab.mercurial-scm.org/D4704
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 19 Sep 2018 11:22:56 -0700] rev 39844
filelog: stop proxying checksize() (API)
This was only used by verify code. And the check using it is now
implemented as part of verifyintegrity(). The method is unused
and is revlog-centric, which means it isn't appropriate for the file
storage interface. So remove it.
Differential Revision: https://phab.mercurial-scm.org/D4703
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 19 Sep 2018 11:20:02 -0700] rev 39843
filelog: remove version attribute (API)
This was only used by verify code. The check it was used for is now
implemented as part of the verifyintegrity() implementation. The
attribute is now unused, is revlog-specific, and isn't appropriate
to be exposing on the file storage interface. So drop it.
Differential Revision: https://phab.mercurial-scm.org/D4702
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 19 Sep 2018 11:17:28 -0700] rev 39842
verify: start to abstract file verification
Currently, the file storage interface has a handful of attributes
that are exclusively or near-exclusively used by repo verification
code. In order to support verification on non-revlog/alternate
storage backends, we'll need to abstract verification so it can
be performed in a storage-agnostic way.
This commit starts that process.
We establish a new verifyintegrity() method on revlogs and expose
it to the file storage interface. Most of verify.verifier.checklog()
has been ported to this new method.
We need a way to represent verification problems. So we invent an
interface to represent a verification problem, invent a revlog type
to implement that interface, and use it.
The arguments to verifyintegrity() will almost certainly change in
the future, once more functionality is ported from the verify code.
And the "revlogv1" version check is very hacky. (The code in verify
is actually buggy because it is comparing the full 32-bit header
integer instead of just the revlog version short. I'll likely fix
this in a subsequent commit.)
Differential Revision: https://phab.mercurial-scm.org/D4701
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 08:58:57 -0700] rev 39841
unionrepo: remove _constructmanifest()
It is unused after a previous refactor. Spotted in D4641.
Differential Revision: https://phab.mercurial-scm.org/D4700
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 08:46:56 -0700] rev 39840
merge with stable
Martin von Zweigbergk <martinvonz@google.com> [Tue, 25 Sep 2018 08:53:20 -0700] rev 39839
encoding: remove unnecessary lambdas from _encodingfixers
They have been unnecessary since 1a3a08b5d4d5 (encoding: remove
workaround for locale.getpreferredencoding(), 2017-05-13). Also rename
the variable since "fixer" sounds like a function.
Differential Revision: https://phab.mercurial-scm.org/D4743
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 25 Sep 2018 18:59:04 -0700] rev 39838
py3: cast exception to bytes
In order to appease Python 3.
Differential Revision: https://phab.mercurial-scm.org/D4733
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 25 Sep 2018 09:11:56 -0700] rev 39837
py3: cast exception to bytes
This was raising a TypeError on Python 3 and preventing most (all?)
wireprotov2 commands from working.
Differential Revision: https://phab.mercurial-scm.org/D4731
Matt Harbison <matt_harbison@yahoo.com> [Tue, 25 Sep 2018 23:25:36 -0400] rev 39836
py3: remove a couple of superfluous calls to pycompat.rapply()
These places can only be strings.
Matt Harbison <matt_harbison@yahoo.com> [Tue, 25 Sep 2018 22:11:17 -0400] rev 39835
py3: byteify an inline python test extension
# skip-blame for b'' prefixing
Matt Harbison <matt_harbison@yahoo.com> [Tue, 25 Sep 2018 21:39:42 -0400] rev 39834
py3: conditionalize access to socketserver.ForkingMixIn
This is no longer exported on platforms that don't support forking, as of 3.6.
https://github.com/python/cpython/commit/aadff9bea61a2fc9f4cf0f213f0ee50fc54d6574
Matt Harbison <matt_harbison@yahoo.com> [Tue, 25 Sep 2018 22:46:18 -0400] rev 39833
convert: fix a file descriptor leak
test-check-code flagged this after I changed this line for something unrelated.
Yuya Nishihara <yuya@tcha.org> [Wed, 26 Sep 2018 20:33:09 +0900] rev 39832
merge with stable
Yuya Nishihara <yuya@tcha.org> [Tue, 25 Sep 2018 22:19:40 +0900] rev 39831
revlog: catch more specific exception in shortest()
Since revlog._partialmatch() catches RevlogError coming from cext and
re-raises AmbiguousPrefixLookupError, catching RevlogError here seems
less correct.
Differential Revision: https://phab.mercurial-scm.org/D4735
Matt Harbison <matt_harbison@yahoo.com> [Mon, 24 Sep 2018 22:32:30 -0400] rev 39830
py3: update missing module list in test-check-py3-compat.t for Windows
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 20:31:42 -0700] rev 39829
py3: add b'' prefixes to wire protocol test
Otherwise the CBOR encoder fails on Python 3 due to lacking support
for encoding str/unicode.
# skip-blame just some b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D4734
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 20:17:42 -0700] rev 39828
py3: use pycompat.strkwargs()
Otherwise we get an error attempting to dispatch a command with
arguments because we're passing a dict with bytes keys.
Differential Revision: https://phab.mercurial-scm.org/D4732
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 20:10:01 -0700] rev 39827
py3: ensure _start_response() is called with system string
This was preventing HTTP 500's from being sent in Python 3.
Differential Revision: https://phab.mercurial-scm.org/D4730
Matt Harbison <matt_harbison@yahoo.com> [Sun, 23 Sep 2018 00:47:04 -0400] rev 39826
py3: convert arguments, cwd and env to native strings when spawning subprocess
This keeps Windows happy.
Matt Harbison <matt_harbison@yahoo.com> [Fri, 21 Sep 2018 21:14:27 -0400] rev 39825
py3: apply byteskwargs to contrib/perf
Matt Harbison <matt_harbison@yahoo.com> [Fri, 21 Sep 2018 20:28:00 -0400] rev 39824
py3: un-byteify strings around os.system() and os.devnull in contrib/perf
Matt Harbison <matt_harbison@yahoo.com> [Fri, 21 Sep 2018 20:16:13 -0400] rev 39823
py3: work around the lack of sys.maxint in contrib/perf
Matt Harbison <matt_harbison@yahoo.com> [Fri, 21 Sep 2018 20:13:14 -0400] rev 39822
py3: switch contrib/perf from xrange to pycompat.xrange
Matt Harbison <matt_harbison@yahoo.com> [Fri, 21 Sep 2018 20:10:36 -0400] rev 39821
py3: handle sysstr conversion around get/set attr in contrib/perf
Matt Harbison <matt_harbison@yahoo.com> [Fri, 21 Sep 2018 20:03:07 -0400] rev 39820
py3: proxy posixfile objects to re-add a useful 'name' attribute on Windows
This file object is used in the vfs layer, so there are many errors like this:
...
File "mercurial\localrepo.py", line 2569, in savecommitmessage
return self.pathto(fp.name[len(self.root) + 1:])
TypeError: 'int' object is not subscriptable
It looks like the 'name' value is actually the fileno() value, and the
documentation says the name parameter to PyFile_FromFd() is ignored. [1] I
tried just assigning the attribute after osutil.posixfile() returns, but that
crashes saying that it's read-only.
[1] https://docs.python.org/3.6/c-api/file.html
Matt Harbison <matt_harbison@yahoo.com> [Sun, 23 Sep 2018 22:36:44 -0400] rev 39819
py3: don't use os.getcwdb() on Windows to avoid DeprecationWarnings
See also ac32685011a3.
Matt Harbison <matt_harbison@yahoo.com> [Fri, 21 Sep 2018 19:48:23 -0400] rev 39818
py3: rename pycompat.getcwd() to encoding.getcwd() (API)
We need to avoid os.getcwdb() on Windows to avoid DeprecationWarnings, and we
need encoding.strtolocal() to encode the result of os.getcwd().
Augie Fackler <augie@google.com> [Mon, 24 Sep 2018 22:46:22 -0400] rev 39817
py3: whitelist two more passing tests
Caught by the ratchet, but initially only the non-legacy path of
test-clone-uncompressed.t was passing. That required the preceding
patch.
Differential Revision: https://phab.mercurial-scm.org/D4729
Augie Fackler <augie@google.com> [Mon, 24 Sep 2018 22:45:32 -0400] rev 39816
keepalive: be more careful about self._rbuf when calling super impls
In Python 3, HTTPResponse implements read() in terms of readinto(),
which was calling back into our readinto(), which duplicates
self._rbuf if it's not empty. Before calling into super's read(),
ensure self._rbuf is empty.
Inheritance is bad, and undocumented self-use of your public API is
one of many reasons.
Differential Revision: https://phab.mercurial-scm.org/D4728
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 17 Sep 2018 11:50:59 -0700] rev 39815
wireprotov2: teach changesetdata to fetch ancestors until depth
For shallow clone, it is useful to specify a starting node and tell
the server to send up to N ancestors from that starting point. This
enables the server to perform the DAG walk without the client having
to discover the base/stop node(s) first.
This commit implements support for said queries on the changesetdata
command.
Differential Revision: https://phab.mercurial-scm.org/D4621
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 20 Sep 2018 12:57:23 -0700] rev 39814
wireprotov2: allow multiple fields to follow revision maps
The *data wire protocol commands emit a series of CBOR values.
Because revision/delta data may be large, their data is emitted
outside the map as a top-level bytestring value.
Before this commit, we'd emit a single optional bytestring
value after the revision descriptor map. This got the job done.
But it was limiting in that we could only send a single field.
And, it required the consumer to know that the presence of a
key in the map implied the existence of a following bytestring
value.
This commit changes the encoding strategy so top-level bytestring
values in the stream are explicitly denoted in a "fieldsfollowing"
key. This key contains an array defining what fields that follow
and the expected size of each field.
By defining things this way, we can easily send N bytestring
values without any ambiguity about their order. In addition,
clients only need to know how to parse ``fieldsfollowing`` to
know if extra values are present.
Because this breaks backwards compatibility, we've bumped the version
number of the wire protocol version 2 API endpoint.
Differential Revision: https://phab.mercurial-scm.org/D4620
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 17 Sep 2018 11:54:00 -0700] rev 39813
wireprotov2: advertise set of valid values for requestable fields
changesetdata, manifestdata, and filedata all allow the caller to
specify what data fields to request.
Data fields are extensible and may evolve over time. In order to
prevent clients from making requests for fields that are not
available, the client needs to know what fields are available.
This commit teaches the server to declare a set of "valid values"
for wire protocol command arguments. That set of values is exposed
in the command's capabilities descriptor. The changesetdata,
manifestdata, and filedata commands all declare their set of
available "fields."
Differential Revision: https://phab.mercurial-scm.org/D4619
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 17 Sep 2018 10:15:27 -0700] rev 39812
wireprotov2: expose rich arguments metadata
Now that we internally store rich metadata about arguments, it makes
sense to make that metadata available to the client. This will allow
clients to validate outgoing command requests before they are sent
over the wire.
Strictly speaking, we should bump the wire protocol version for this
change since it is backwards incompatible. But no client-side code
touches the arguments map and I don't want to incur the work.
Differential Revision: https://phab.mercurial-scm.org/D4618
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 17 Sep 2018 09:49:28 -0700] rev 39811
wireprotov2: advertise recognized path filter prefixes
While the wire protocol doesn't yet support it, we'll eventually
have commands that accept narrow patterns to specify the set of
files relevant to a command.
For security and performance reasons, only specific filter types
are allowed.
This commit teaches the server to advertise the set of allowed
filter types. By doing so, clients can e.g. validate user-specified
patterns against the server's abilities without having to send
a command to retrieve data.
Having the data in the capabilities data structure will also serve
as a check against unwanted BC.
Differential Revision: https://phab.mercurial-scm.org/D4616
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 30 Aug 2018 17:43:47 -0700] rev 39810
wireprotov2: declare command arguments richly
Previously, we declared command arguments with an example of
their value. After this commit, we declare command arguments
as a dict of metadata. This allows us to define the value
type, whether the argument is required, and provide default
values. This in turn allows us to have nice things, such as
less boilerplate code in individual commands for validating
input and assigning default values. It should also make command
behavior more consistent as a result.
Test output changed slightly because I realized that the "fields"
argument wasn't being consistently defined as a set. Oops!
Other test output changed because of slight differences in code
performing type validation.
Differential Revision: https://phab.mercurial-scm.org/D4615
Yuya Nishihara <yuya@tcha.org> [Wed, 26 Sep 2018 21:24:14 +0900] rev 39809
chgserver: do not send system() back to client if stdio redirected (issue5992)
As the chg client doesn't know server-side stdio redirection, the server
shouldn't upcall on "runsystem" request if the stdio streams are redirected.
This patch teaches ui to remember the redirection flag, which is updated by
the caller right now. Future patches (for default) will add ui methods to
manage this flag internally.
Yuya Nishihara <yuya@tcha.org> [Wed, 26 Sep 2018 21:21:05 +0900] rev 39808
chgserver: update comment describing when to fall back to core _runsystem()
The condition "output needs to be captured" was moved to the core ui, and
removed at fbce78c58f1e "chg: refactor ui.system() to be partly overridden."
The next patch will add one more "if" to address the issue 5992.
Yuya Nishihara <yuya@tcha.org> [Wed, 26 Sep 2018 20:53:59 +0900] rev 39807
procutil: compare fd number to see if stdio protection is needed (issue5992)
When I wrote this function for commandserver at 69f86b937035, testing object
identity was suffice, and I was sloppy enough not to compare fileno() values.
However, it doesn't work in chg session because chgserver reopens stdio to
apply new buffering mode.
This patch partially fixes the issue 5992. Still we have another problem in
chgui._runsystem().
Yuya Nishihara <yuya@tcha.org> [Tue, 25 Sep 2018 23:06:02 +0900] rev 39806
test-ssh: show that stdio redirection doesn't work with chg
Running tests with --chg doesn't mean all hg invocations are replaced by
chg. This patch explicitly adds the test for "chg serve --stdio", which does
weird stdio dance.
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Tue, 25 Sep 2018 16:32:38 -0400] rev 39805
revset: make heads(commonancestors(x + x^)) be x^, not x
Differential Revision: https://phab.mercurial-scm.org/D4742
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Tue, 25 Sep 2018 16:29:39 -0400] rev 39804
revset: add tests of heads(commonancestors(..))
The second-to-last one shows the same bug as commonancestors(..): the
result should be 8, not 9.
Differential Revision: https://phab.mercurial-scm.org/D4741
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Tue, 25 Sep 2018 16:18:43 -0400] rev 39803
revset: reword commonancestor()'s help
The new version seems a bit more consistent with other doc comments,
and feels clearer to me (doesn't explain "commonancestors(set)" as
"common ancestors of set").
Differential Revision: https://phab.mercurial-scm.org/D4740
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Tue, 25 Sep 2018 16:14:57 -0400] rev 39802
revset: remove part of the commonancestors() comment
The reason is that:
- it shows up in "hg help revset", but it doesn't look like
documentation targeted at users
- it doesn't make sense to me: it doesn't say what happens with < 2
revisions, and is not quite right because my understanding is that
this revset was created precisely because "::x and ::y" was not
quite the same (when x and y don't evaluate to singletons).
Differential Revision: https://phab.mercurial-scm.org/D4739
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Tue, 25 Sep 2018 16:05:21 -0400] rev 39801
revset: make commonancestors(x + x^) be ::(x^), not ::x
Differential Revision: https://phab.mercurial-scm.org/D4738
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Tue, 25 Sep 2018 15:27:41 -0400] rev 39800
revset: add test demonstrating a bug with commonancestor()
Specifically, 9 is clearly not in "::8 and ::9".
Differential Revision: https://phab.mercurial-scm.org/D4737
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Tue, 25 Sep 2018 16:03:14 -0400] rev 39799
revset: fix commonancestor test so it demonstrates correct behavior
The problem is that hg log -r 'head()' is every changeset in the
repository, because in this test repository, every changeset has a
different branch. The author probably assumed all commits were on the
default branch, and that they were getting topological heads, 7 and 9.
As a result, this test was showing that the common ancestors of
0:9 are 0+1+2+4, which is not correct (next commit will test this).
Differential Revision: https://phab.mercurial-scm.org/D4736
Yuya Nishihara <yuya@tcha.org> [Sat, 15 Sep 2018 17:26:23 +0900] rev 39798
annotate: pass in wdir rev and node to formatter (BC)
This is a part of the unification series. The 'ff..' hash is preferred over
None as it is a valid revision specifier.
https://www.mercurial-scm.org/wiki/GenericTemplatingPlan#Sanity_check_output
Yuya Nishihara <yuya@tcha.org> [Sat, 15 Sep 2018 17:26:22 +0900] rev 39797
annotate: pass around full hex node until formatting plain output
In short, this patch moves h[:12] from hexfn() to formathex() so that
formathex() can test if h is the wdirhex or not. This helps switching the
wdir value to wdirrev/wdirhex. See the next patch.
Yuya Nishihara <yuya@tcha.org> [Sat, 15 Sep 2018 17:26:21 +0900] rev 39796
log: fill in pseudo rev and node as wdir() manifest identifiers
While we'll never support such identifiers to look up the manifest,
this behavior seems more consistent.
Yuya Nishihara <yuya@tcha.org> [Sat, 15 Sep 2018 17:26:20 +0900] rev 39795
log: have changesetformatter fill in wdir() rev and node (BC)
This matches the behavior of the default template keywords. wdir() support
is still experimental so we can change the output.
Yuya Nishihara <yuya@tcha.org> [Sun, 23 Sep 2018 16:11:01 +0900] rev 39794
hgweb: use scmutil.binnode() to translate None to wdir hash (issue5988)
I left some of ctx.node() calls unchanged as they seemed unlikely to be
workingctx, or passed to diff functions where None is the default value.
Note that a None revision can also cause a similar problem, but I'm not sure
if we can simply bulk-replace ctx.rev() with scmutil.intrev(ctx) as there's
large hole between tip revision and wdir revision. If such pair were passed
in to xrange() for example, we would waste CPU time.
Yuya Nishihara <yuya@tcha.org> [Sun, 23 Sep 2018 16:15:48 +0900] rev 39793
hgweb: register web.static to the config table
Otherwise we would got a develwarn.
Yuya Nishihara <yuya@tcha.org> [Sun, 23 Sep 2018 17:01:26 +0900] rev 39792
help: document web.static config
Spotted by check-config.py.
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 24 Sep 2018 00:00:04 +0300] rev 39791
py3: use pycompat.strkwargs() in contrib/simplemerge
Differential Revision: https://phab.mercurial-scm.org/D4699
Pulkit Goyal <pulkit@yandex-team.ru> [Sun, 23 Sep 2018 23:59:35 +0300] rev 39790
py3: add b'' prefixes in contrib/simplemerge
They were added using byteify-strings.py.
# skip-blame because just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D4698
Pulkit Goyal <pulkit@yandex-team.ru> [Sun, 23 Sep 2018 23:39:07 +0300] rev 39789
tests: use assertTrue() instead of assert_() in test-bdiff.py
The later is deprecated in Python 3.
This patch also makes the test pass on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D4697
Pulkit Goyal <pulkit@yandex-team.ru> [Sun, 23 Sep 2018 23:35:38 +0300] rev 39788
py3: use '%d' for integers instead of '%s'
Python 3 does not allow using "%s" for integers.
This makes the test works on Python 3. The test still does not pass because
there is a deprecation warning.
Differential Revision: https://phab.mercurial-scm.org/D4696
Pulkit Goyal <pulkit@yandex-team.ru> [Sun, 23 Sep 2018 23:34:22 +0300] rev 39787
py3: add b'' prefixes in tests/test-bdiff.py
This makes the test close to passing
# skip-blame because just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D4695