Stanislau Hlebik <stash@fb.com> [Mon, 13 Feb 2017 02:31:56 -0800] rev 30906
localrepo: avoid unnecessary sorting
headrevs output already sorted, no need to sort it again.
Stanislau Hlebik <stash@fb.com> [Mon, 13 Feb 2017 02:26:18 -0800] rev 30905
localrepo: cache self.changelog in local variable
Repeated self.changelog lookups can incur overhead. Let's cache it in the
separate variable.
Martin von Zweigbergk <martinvonz@google.com> [Tue, 07 Feb 2017 13:11:30 -0800] rev 30904
destutil: remove dead code about non-linear updates
IIUC, the non-linear updates no longer happen by default since
6b1fc09c699a (update: change default destination to tipmost descendant
(issue4673) (BC), 2016-02-02), and it was only if they happened by
default that we used to error out, so there is no longer a need to
handle this case.
Martin von Zweigbergk <martinvonz@google.com> [Thu, 09 Feb 2017 09:55:31 -0800] rev 30903
update: fix typo/stale comment to match code
The comment about "obsolete.background" seems to have been about
obsolete.foreground ever since it was introduced in a59e575c6ff8
(update: allow dirty update to foreground (successors), 2013-04-16),
so let's change it.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 08 Feb 2017 23:03:33 -0800] rev 30902
merge: remove unused handling of default destination in merge.update()
As far as I can tell, all the callers of merge.update() have been
migrated over to use destutil to find the default destination when the
revision was not specified. So it's time to delete the code for
handling a node value of None. Let's add an assertion that node is not
None, so any extensions relying on it will not silently misbehave.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 08 Feb 2017 14:49:37 -0800] rev 30901
update: localize logic around which ancestor to use
The merge code works by applying the changes between an ancestor and
the working copy onto the destination. To achieve an update, it sets
the ancestor to be the parent of the working copy. To achieve a clean
update (update -C), it sets the ancestor to be the working copy itself
(so there are no changes to carry over). The logic for this was spread
out a bit. Let's move it all to one place so it's easier to follow the
reason for it. Also add some documentation.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 08 Feb 2017 22:12:27 -0800] rev 30900
tests: add test for updating to null revision
While working on merge.py, I realized that we don't (as far as I could
tell) have any tests for updating to the null revision with a dirty
working copy. This adds some simple tests for that.
Martin von Zweigbergk <martinvonz@google.com> [Fri, 10 Feb 2017 15:26:03 -0800] rev 30899
import: mention "stdin" (abbreviated) and add example
I actually didn't even think it was possible because I searched the
help text for "stdin", and didn't even think of searching for
"standard input". Let's mention the abbreviated form too to help
others like me. (When importing from stdin, we actually print a
message saying "applying patch from stdin".)
This patch also adds an example showing how to import from stdin.
Martin von Zweigbergk <martinvonz@google.com> [Thu, 09 Feb 2017 09:32:25 -0800] rev 30898
merge: print status message before launching external merge tool
It seems somewhat common that people run into a merge conflict and
don't notice the launched merge tool, and instead they think hg just
hung. Let's print a message for each file that we launch a GUI merge
tool for.
Simon Farnsworth <simonfar@fb.com> [Wed, 08 Feb 2017 07:44:10 -0800] rev 30897
pager: exit cleanly on SIGPIPE (BC)
Changeset aaa751585325 removes SIGPIPE handling completely. This is wrong,
as it means that Mercurial does not exit when the pager does. Instead, raise
SignalInterrupt when SIGPIPE happens with a pager attached, to trigger the
normal exit path.
This will cause "killed!" to be printed to stderr (hence the BC warning),
but in the normal pager use case (where the pager gets both stderr and
stdout), this message is lost as we only get SIGPIPE when the pager quits.
Jun Wu <quark@fb.com> [Fri, 10 Feb 2017 04:09:06 -0800] rev 30896
runtests: catch EPROTONOSUPPORT in checkportisavailable
This is a follow-up of "runtests: check ports on IPv6 address". On some
platforms, "socket.AF_INET6" exists while that does not necessarily mean the
platform support IPv6 - when initializing a socket using "socket.socket", it
could fail with EPROTONOSUPPORT. So treat that as "Port unavailable".
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 07 Feb 2017 23:24:47 -0800] rev 30895
zstd: vendor python-zstandard 0.7.0
Commit 3054ae3a66112970a091d3939fee32c2d0c1a23e from
https://github.com/indygreg/python-zstandard is imported without
modifications (other than removing unwanted files).
The vendored zstd library within has been upgraded from 1.1.2 to
1.1.3. This version introduced new APIs for threads, thread
pools, multi-threaded compression, and a new dictionary
builder (COVER). These features are not yet used by
python-zstandard (or Mercurial for that matter). However,
that will likely change in the next python-zstandard release
(and I think there are opportunities for Mercurial to take
advantage of the multi-threaded APIs).
Relevant to Mercurial, the CFFI bindings are now fully
implemented. This means zstd should "just work" with PyPy
(although I haven't tried). The python-zstandard test suite also
runs all tests against both the C extension and CFFI bindings to
ensure feature parity.
There is also a "decompress_content_dict_chain()" API. This was
derived from discussions with Yann Collet on list about alternate
ways of encoding delta chains.
The change most relevant to Mercurial is a performance enhancement in
the simple decompression API to reuse a data structure across
operations. This makes decompression of multiple inputs significantly
faster. (This scenario occurs when reading revlog delta chains, for
example.)
Using python-zstandard's bench.py to measure the performance
difference...
On changelog chunks in the mozilla-unified repo:
decompress discrete decompress() reuse zctx
1.262243 wall; 1.260000 CPU; 1.260000 user; 0.000000 sys 170.43 MB/s (best of 3)
0.949106 wall; 0.950000 CPU; 0.950000 user; 0.000000 sys 226.66 MB/s (best of 4)
decompress discrete dict decompress() reuse zctx
0.692170 wall; 0.690000 CPU; 0.690000 user; 0.000000 sys 310.80 MB/s (best of 5)
0.437088 wall; 0.440000 CPU; 0.440000 user; 0.000000 sys 492.17 MB/s (best of 7)
On manifest chunks in the mozilla-unified repo:
decompress discrete decompress() reuse zctx
1.367284 wall; 1.370000 CPU; 1.370000 user; 0.000000 sys 274.01 MB/s (best of 3)
1.086831 wall; 1.080000 CPU; 1.080000 user; 0.000000 sys 344.72 MB/s (best of 3)
decompress discrete dict decompress() reuse zctx
0.993272 wall; 0.990000 CPU; 0.990000 user; 0.000000 sys 377.19 MB/s (best of 3)
0.678651 wall; 0.680000 CPU; 0.680000 user; 0.000000 sys 552.06 MB/s (best of 5)
That should make reads on zstd revlogs a bit faster ;)
# no-check-commit
Augie Fackler <augie@google.com> [Thu, 09 Feb 2017 21:44:32 -0500] rev 30894
tests: exclude python-zstandard from pyflakes analysis
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 07 Feb 2017 23:25:37 +0530] rev 30893
py3: fix the way we produce bytes list in store.py
bytes(range(127)) does not produce a list whereas we need a list. This patch
fixes that.
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 07 Feb 2017 22:47:24 +0530] rev 30892
py3: convert os.__file__ to bytes
os.__file__ returns unicode path on Python 3. We need to have bytespath. This
patch uses pycompat.fsencode() to encode unicode path to bytes path.
Jun Wu <quark@fb.com> [Wed, 08 Feb 2017 14:45:30 -0800] rev 30891
commandserver: handle backlog before exiting
Previously, when a chg server is exiting, it does not handle connected
clients so clients may get ECONNRESET and crash:
1. client connect() # success
2. server shouldexit = True and exit
3. client recv() # ECONNRESET
d7875bfbfccb makes this race condition easier to reproduce if a lot of short
chg commands are started in parallel.
This patch fixes the above issue by unlinking the socket path to stop
queuing new connections and processing all pending connections before exit.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sat, 11 Feb 2017 00:23:55 +0900] rev 30890
misc: replace domain of mercurial-devel ML address by mercurial-scm.org
This patch also adds new check-code.py pattern to detect invalid usage
of "mercurial-devel@selenic.com".
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sat, 11 Feb 2017 00:23:55 +0900] rev 30889
i18n: update Report-Msgid-Bugs-To property of *.po files
This patch replaces domain of mercurial-devel ML address by
mercurial-scm.org for "Report-Msgid-Bugs-To" property of each *.po
files.
This avoids releasing 4.1.1 with invalid "Report-Msgid-Bugs-To"
in *.mo file, if corresponded *.po file isn't msgmerge-ed with recent
hg.pot by translator.
These *.po files aren't covered by check-code.py pattern newly added
in subsequent patch, because it ignores them.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sat, 11 Feb 2017 00:23:53 +0900] rev 30888
misc: replace domain of mercurial ML address by mercurial-scm.org
This patch also adds new check-code.py pattern to detect invalid usage
of "mercurial@selenic.com".
Change for test-convert-tla.t is tested, but similar change for almost
same test-convert-baz.t isn't yet tested actually, because I couldn't
find out the way to get "GNU Arch baz client".
AFAIK, buildbot skips test-convert-baz.t, too. Does anybody have
appropriate environment for testing?
Jun Wu <quark@fb.com> [Wed, 08 Feb 2017 14:37:38 -0800] rev 30887
commandserver: prevent unlink socket twice
This patch changes unixforkingservice so it only calls
`self._servicehandler.unlinksocket(self.address)` at most once.
This is needed by the next patch.
Jun Wu <quark@fb.com> [Thu, 09 Feb 2017 05:57:54 -0800] rev 30886
runtests: check ports on IPv6 address
Previously, checkportisavailable only checks ports on the IPv4 address. This
patch makes it check IPv6 as well. It'll be useful if "localhost" does not
have an IPv4 address, or its IPv4 address does not exist somehow.
Simon Farnsworth <simonfar@fb.com> [Wed, 08 Feb 2017 08:08:41 -0800] rev 30885
zeroconf: fail nicely on IPv6 only system
zeroconf only knows how to deal with IPv4; I develop on a system where the only
IPv4 address is 127.0.0.1.
Teach zeroconf to ignore IPv6 addresses when looking for plausible IPv4
connectivity.
Jun Wu <quark@fb.com> [Mon, 06 Feb 2017 17:01:06 -0800] rev 30884
chg: verify XDG_RUNTIME_DIR
According to the specification [1], $XDG_RUNTIME_DIR should be ignored
unless:
The directory MUST be owned by the user, and he MUST be the only one
having read and write access to it. Its Unix access mode MUST be 0700.
This patch adds a check and ignores it if it does not meet part of the
criteria.
[1]: https://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
Yedidya Feldblum <yfeldblum@fb.com> [Sat, 21 Jan 2017 14:43:13 -0800] rev 30883
check-code: permit functools.reduce
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 04 Feb 2017 08:47:46 -0800] rev 30882
perf: split obtaining chunks from decompression
Previously, the code was similar to what revlog._chunks() was doing,
which took a raw data segment and delta chain, obtained buffers for
the raw revlog chunks within, and decompressed them.
This commit splits the "get raw chunks" action from "decompress." The
goal of this change is to more accurately measurely decompression
performance.
On a ~50k deltachain for a manifest in mozilla-central:
! full
! wall 0.430548 comb 0.440000 user 0.410000 sys 0.030000 (best of 24)
! deltachain
! wall 0.016053 comb 0.010000 user 0.010000 sys 0.000000 (best of 181)
! read
! wall 0.008078 comb 0.010000 user 0.000000 sys 0.010000 (best of 362)
! rawchunks
! wall 0.033785 comb 0.040000 user 0.040000 sys 0.000000 (best of 100)
! decompress
! wall 0.327126 comb 0.320000 user 0.320000 sys 0.000000 (best of 31)
! patch
! wall 0.032391 comb 0.030000 user 0.030000 sys 0.000000 (best of 100)
! hash
! wall 0.012587 comb 0.010000 user 0.010000 sys 0.000000 (best of 233)
Yuya Nishihara <yuya@tcha.org> [Sun, 16 Oct 2016 17:28:51 +0900] rev 30881
smartset: move set classes and related functions from revset module (API)
These classes are pretty large and independent from revset computation.
2961 mercurial/revset.py
973 mercurial/smartset.py
3934 total
revset.prettyformatset() is renamed to smartset.prettyformat(). Smartset
classes are aliased since they are quite common in revset.py.
Yuya Nishihara <yuya@tcha.org> [Wed, 25 Jan 2017 22:39:17 +0900] rev 30880
help: test if "hg help TOPIC" reference is valid
It's quite easy to make a reference invalid by mistake.
Yuya Nishihara <yuya@tcha.org> [Wed, 25 Jan 2017 22:35:40 +0900] rev 30879
help: uppercase command placeholder
'command' isn't a valid help topic but a placeholder text. Make it upper
case to avoid confusion.
Yuya Nishihara <yuya@tcha.org> [Sun, 05 Feb 2017 18:57:19 +0900] rev 30878
help: show section that couldn't be found
For better error indication.
Augie Fackler <augie@google.com> [Fri, 03 Feb 2017 16:01:19 -0500] rev 30877
cmdutil: remove forwarding methods per deprecation policy
Simon Farnsworth <simonfar@fb.com> [Fri, 03 Feb 2017 15:10:27 -0800] rev 30876
util: always force line buffered stdout when stdout is a tty (BC)
pager replaced stdout with a line buffered version to work around glibc
deciding on a buffering strategy on the first write to stdout. This is going
to make my next patch hard, as replacing stdout will make tracking time
spent blocked on it more challenging.
Move the line buffering requirement to util.py, and remove it from pager.
This means that the abuse of ui.formatted=True and pager set to cat or equivalent
no longer results in a line-buffered output to a pipe, hence (BC), although
I don't expect anyone to be affected
Stanislau Hlebik <stash@fb.com> [Thu, 02 Feb 2017 02:56:38 -0800] rev 30875
localrepo: avoid unnecessary conversion from node to rev
changelog.heads() first calls headrevs then converts them to nodes.
localrepo.heads() then sorts them using self.changelog.rev function and makes
useless conversion back to revs. Instead let's call changelog.headrevs() from
localrepo.heads(), sort the output and then convert to nodes. Because headrevs
does not support start parameter this optimization only works if start is None.
Anton Shestakov <av6@dwimlabs.net> [Sat, 04 Feb 2017 20:29:34 +0800] rev 30874
debian: update copyright years
Anton Shestakov <av6@dwimlabs.net> [Sat, 04 Feb 2017 20:29:13 +0800] rev 30873
debian: update mailing list address
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 02 Feb 2017 14:19:48 +0100] rev 30872
bundle2: implement a basic __repr__ for bundle2 part
We display basic data as the part id and part type. This make debugging bundle2
related code friendlier.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 02 Feb 2017 11:03:41 +0100] rev 30871
bundle2: drop an outdated comment
The function is no longer in "early" stage and have been used in production for
years. We can probably drop that part of the docstring...
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 02 Feb 2017 10:53:55 +0100] rev 30870
unbundle: swap conditional branches for clarity
This is a small style update for clarity. The previous situation was:
if foo:
50 lines
else:
2 lines
In such case I tend to invert these to get the simpler branch out of the way
earlier:
if not foo:
2 lines
else:
50 lines
This makes the conditional and various alternatives fit on the same screen,
simpler to read overall.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 02 Feb 2017 10:55:38 +0100] rev 30869
unbundle: add a small comment to tag the bundle1 case as such
This makes the code clearer to understand for someone new to it (or rusted)
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 02 Feb 2017 10:51:04 +0100] rev 30868
unbundle: add a small comment to clarify the 'check_heads' call
Bundle2 has its own mechanisms to check for heads (and other) changes, so push
using bundle2 is relying on the "check:heads" bundle part of unbundle and the
'check_heads' call is not checking anything. We add a small comment to make
this clearer.
Simon Farnsworth <simonfar@fb.com> [Thu, 02 Feb 2017 11:17:36 -0800] rev 30867
pager: don't terminate with extreme prejudice on SIGPIPE (BC)
The default SIGPIPE handler causes Mercurial to exit immediately, without
running any Python cleanup code (except and finally blocks, atexit handlers
etc). This creates problems if you want to do something at exit.
If we need a different exit code for broken pipe from pager, then we should
code that ourselves in Python; this appears to have been cargo-culted from
the fork implementation of pager that's no longer used, where it was needed
to stop Broken Pipe errors appearing on the user's terminal.
Martin von Zweigbergk <martinvonz@google.com> [Mon, 23 Jan 2017 10:48:55 -0800] rev 30866
verify: replace _validpath() by matcher
The verifier calls out to _validpath() to check if it should verify
that path and the narrowhg extension overrides _validpath() to tell
the verifier to skip that path. In treemanifest repos, the verifier
calls the same method to check if it should visit a
directory. However, the decision to visit a directory is different
from the condition that it's a matching path, and narrowhg was working
around it by returning True from its _validpath() override if *either*
was true.
Similar to how one can do "hg files -I foo/bar/ -X foo/" (making the
include pointless), narrowhg can be configured to track the same
paths. In that case match("foo/bar/baz") would be false, but
match.visitdir("foo/bar/baz") turns out to be true, causing verify to
fail. This may seem like a bug in visitdir(), but it's explicitly
documented to be undefined for subdirectories of excluded
directories. When using treemanifests, the walk would not descend into
foo/, so verification would pass. However, when using flat manifests,
there is no recursive directory walk and the file path "foo/bar/baz"
would be passed to _validpath() without "foo/" (actually without the
slash) being passed first. As explained above, _validpath() would
return true for the file path and "hg verify" would fail.
Replacing the _validpath() method by a matcher seems like the obvious
fix. Narrowhg can then pass in its own matcher and not have to
conflate the two matching functions (for dirs and files). I think it
also makes the code clearer.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 01 Feb 2017 08:47:27 -0800] rev 30865
rebase: fix code comment to refer to right issue (4504, not 4505)
The comment was introduced in 8a544fb645bb (rebase: ensure rebase
revision remains visible (issue4504), 2015-01-27), which mentions the
right issue in the description.
Kevin Bullock <kbullock+mercurial@ringworld.org> [Wed, 01 Feb 2017 11:30:26 -0600] rev 30864
merge with stable
Kevin Bullock <kbullock@ringworld.org> [Wed, 01 Feb 2017 10:19:49 -0600] rev 30863
Added signature for changeset e1526da1e6d8
Kevin Bullock <kbullock@ringworld.org> [Wed, 01 Feb 2017 10:18:59 -0600] rev 30862
Added tag 4.1 for changeset e1526da1e6d8
Kevin Bullock <kbullock+mercurial@ringworld.org> [Wed, 01 Feb 2017 10:15:10 -0600] rev 30861
merge with i18n
Wagner Bruna <wbruna@softwareexpress.com.br> [Wed, 01 Feb 2017 08:47:11 -0200] rev 30860
i18n-pt_BR: synchronized with dfc6663f97ca
Mads Kiilerich <mads@kiilerich.com> [Wed, 01 Feb 2017 02:10:30 +0100] rev 30859
merge: more safe detection of criss cross merge conflict between dm and r
41f6af50c0d8 introduced handling of a crash in this case. A review comment
suggested that it was not entirely obvious that a 'dm' always would have a 'r'
for the source file.
To mitigate that risk, make the code more conservative and make less
assumptions.
Augie Fackler <augie@google.com> [Mon, 30 Jan 2017 18:03:17 -0500] rev 30858
tests: correct (I think) command in test-largefiles-update
When this test was introduced, it used the short-form of all the flags
on this update invocation. I suspect, based on the "start with clean
dirstates" comment and the fact that the no-exec branch of the #if
guard leaves dirstate clean, that this should have been 'update -qCr'
instead of 'update -qcr', but that a bug in largefiles --check
handling left this problem unnoticed.
I'll leave a breadcrumb further up about the current failure mode in
the hopes that we can fix this some day.
This was previously discussed in [0] but the trail in that thread goes
cold after a few replies. Given that this is still a flaky test, that
appears to only be passing by bad fortune, I think it's worth
correcting the code of the test to make a correct assertion, and to
keep track of the suspected bug with some other mechanism than an
invalid test (if we had support for "expected failure" blocks this
might be a worthwhile use of them?).
0: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-October/089501.html
Augie Fackler <augie@google.com> [Mon, 30 Jan 2017 17:57:21 -0500] rev 30857
tests: expand flags to long form in test-largefiles-update.t
I spent some time confused by this test. I'm pretty sure that this
line intends to be cleaning the dirstate, not checking that it's clean
before updating: the preceding #if block leaves the dirstate clean in
the noexec case, and dirty in the exec case, so we can't expect
consistent behavior across that platform variation. A subsequent patch
will modify this command to use --clean instead of --check.
I'll elaborate in that patch about the hypothetical bug here.
Mads Kiilerich <mads@kiilerich.com> [Tue, 31 Jan 2017 03:25:59 +0100] rev 30856
merge: fix crash on criss cross merge with dir move and delete (issue5020)
Work around that 'dm' in the data model only can have one operation for the
target file, but still can have multiple and conflicting operations on the
source file where the other operation is a 'rm'. The move would thus fail with
'abort: No such file or directory'.
In this case it is "obvious" that the file should be removed, either before or
after moving it. We thus keep the 'rm' of the source file but drop the 'dm'.
This is not a pretty fix but quite "obviously" safe (famous last words...) as
it only touches a rare code path that used to crash. It is possible that it
would be better to swap the files for 'dm' as suggested on
https://bz.mercurial-scm.org/show_bug.cgi?id=5020#c13 but it is not entirely
obvious that it not just would create conflicts on the other file. That can be
revisited later.
Mads Kiilerich <mads@kiilerich.com> [Tue, 31 Jan 2017 03:20:07 +0100] rev 30855
tests: use 'f' in test-merge-criss-cross.t to prepare for recursive dumping
Prepare for adding a test case with files in a directory.
Martin von Zweigbergk <martinvonz@google.com> [Mon, 30 Jan 2017 22:58:56 -0800] rev 30854
util: make sortdict.keys() return a copy
dict.keys() is documented to return a copy, so it's surprising that
sortdict.keys() did not. I noticed this because we have an extension
that calls readlocaltags(). That method tries to remove any tags that
point to non-existent revisions (most likely stripped). However, since
it's unintentionally working on the instance it's modifying, it
sometimes fails to remove tags when there are multiple bad tags in a
row. This was not caught because localrepo.tags() does an additional
layer of filtering.
sortdict is also used in other places, but I have not checked whether
its keys() and/or __delitem__() methods are used there.
Yuya Nishihara <yuya@tcha.org> [Mon, 30 Jan 2017 22:50:20 +0900] rev 30853
test-highlight: add normalization rule for Pygments 2.2
The test failed on Debian sid because of new class="vm".
Danek Duvall <danek.duvall@oracle.com> [Sun, 29 Jan 2017 12:40:56 -0800] rev 30852
tests: account for different newline behavior between Solaris and GNU grep
GNU grep, when emitting a matching line that doesn't have a terminating
newline, will add an extra newline. Solaris grep passes the original line
through without the newline. This causes differences in test output when
looking at the last line of the output of get-with-headers.py, which
doesn't usually emit (and certainly doesn't guarantee) a terminating
newline.
Both grep implementations succeed in matching the requested pattern,
though, so rely on specifying the full pattern on grep's commandline
instead of expecting it in the output, and send the output to /dev/null.
Augie Fackler <augie@google.com> [Fri, 20 Jan 2017 10:17:34 -0500] rev 30851
tests: also allow "Protocol not supported" in test-http-proxy error
I've seen this in a (misconfigured) FreeBSD jail which has ::1 as an
entry for localhost, but IPv6 support is disabled in the jail. It took
me months to figure out what was going on (and I only figured it out
when tinyproxy.py got confused by similar IPv4-level misconfiguration
of the localhost domain in /etc/hosts.)
I don't feel strongly about this patch: on the one hand, it's papering
over a host-level misconfiguration, but on the other it avoids some
weird and hard to diagnose problems that can occur in weirdly
restricted environments.
Yuya Nishihara <yuya@tcha.org> [Fri, 20 Jan 2017 21:33:18 +0900] rev 30850
revset: prevent using outgoing() and remote() in hgweb session (BC)
outgoing() and remote() may stall for long due to network I/O, which seems
unsafe per definition, "whether a predicate is safe for DoS attack." But I'm
not 100% sure about this. If our concern isn't elapsed time but CPU resource,
these predicates are considered safe. Perhaps that would be up to the
web/application server configuration?
Anyway, outgoing() and remote() wouldn't be useful in hgweb, so I think
it's okay to ban them.
Augie Fackler <augie@google.com> [Thu, 19 Jan 2017 16:23:49 -0500] rev 30849
tests: use an absolute path to get around '..' being invalid on a dead CWD
Only FreeBSD seems to be this picky. Note that this explicit
absolute-path `cd` exposes a defect in the test, in that we end up
still inside the cwd-vanish repository, but that's not a regression in
this change. Since we're in a code freeze, I'm doing the smallest
thing possible to try and fix bugs on FreeBSD, rather than cleaning up
the entire problem. I'll follow up with a more complete fix after the
freeze.
Sean Farley <sean@farley.io> [Wed, 18 Jan 2017 18:25:51 -0800] rev 30848
ui: rename tmpdir parameter to more specific repopath
This was requested by Augie and I agree that repopath is more
descriptive.
Yuya Nishihara <yuya@tcha.org> [Thu, 19 Jan 2017 23:01:32 +0900] rev 30847
pager: wrap _runcommand() no matter if stdout is redirected
We've made chg utilize the common code path implemented in pager.py (by
815e1cefd082 and 493935e0327a), but the chg server does not always start
with a tty. Because of this, uisetup() of the pager extension could be
skipped on the chg server.
Kudos given to Sean Farley for dogfooding new chg and spotting this problem.
Kostia Balytskyi <ikostia@fb.com> [Thu, 19 Jan 2017 09:48:40 -0800] rev 30846
shelve: make unshelve not crash when there are missing files (issue4176)
This patch makes it possible to unshelve while having missing files
in your repo as long as shelved changes don't touch those missing files.
It also makes error message better otherwise.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 18 Jan 2017 22:45:07 -0800] rev 30845
statprof: require input file
statprof has a __main__ handler that allows viewing of previously
written data files. As Yuya pointed out during review, f42cd5434cc2
broke this. This patch fixes that.
Augie Fackler <augie@google.com> [Wed, 18 Jan 2017 23:43:41 -0500] rev 30844
tests: work around FreeBSD's unzip having slightly different output
According to man 1 unzip, this unzip appeared in FreeBSD 8.0. It's
what comes as /usr/bin/unzip, so we may as well cater to it since it's
easy.
Augie Fackler <augie@google.com> [Wed, 18 Jan 2017 23:34:35 -0500] rev 30843
contrib: fix check-commit to not reject commits from `hg sign` and `hg tag`
I'm tired of having a spurious red build every time we do a
release. Fix it once and for all.
Augie Fackler <raf@durin42.com> [Wed, 18 Jan 2017 20:03:00 -0500] rev 30842
Added signature for changeset a1dd2c0c479e
Augie Fackler <raf@durin42.com> [Wed, 18 Jan 2017 20:02:58 -0500] rev 30841
Added tag 4.1-rc for changeset a1dd2c0c479e
Augie Fackler <augie@google.com> [Wed, 18 Jan 2017 11:54:51 -0500] rev 30840
tests: fix up some http tests for no-zstd case
Augie Fackler <augie@google.com> [Wed, 18 Jan 2017 11:43:36 -0500] rev 30839
freeze: merge default into stable for 4.1 code freeze
Sean Farley <sean@farley.io> [Mon, 16 Jan 2017 21:17:39 -0800] rev 30838
patchbomb: add tmpdir parameter to ui.edit call
Sean Farley <sean@farley.io> [Mon, 16 Jan 2017 21:15:57 -0800] rev 30837
histedit: add tmpdir parameter to ui.edit call
Sean Farley <sean@farley.io> [Mon, 16 Jan 2017 21:15:21 -0800] rev 30836
cmdutil: add tmpdir parament to ui.edit calls
Sean Farley <sean@farley.io> [Mon, 16 Jan 2017 21:05:22 -0800] rev 30835
ui: add a parameter to set the temporary directory for edit
Until callsites are updated, this will have no effect. Once callsites
are updated, specifying experimental.editortmpinhg will create editor
temporary files in a subdirectory of .hg, which will make it easier
for tool integrations to determine what repository is in play when
they're asked to edit an hg-related file.
Pulkit Goyal <7895pulkit@gmail.com> [Wed, 18 Jan 2017 03:44:19 +0530] rev 30834
help: update help for `hg update` which was misleading (issue5427)
Matt Harbison <matt_harbison@yahoo.com> [Tue, 17 Jan 2017 23:12:54 -0500] rev 30833
templater: add '{envvars}' to access environment variables
Since the option for ui.exportableenviron is experimental, so is this template
until the underlying API is sorted out.
Matt Harbison <matt_harbison@yahoo.com> [Tue, 17 Jan 2017 23:05:12 -0500] rev 30832
ui: introduce an experimental dict of exportable environment variables
Care needs to be taken to prevent leaking potentially sensitive environment
variables through hgweb, if template support for environment variables is to be
introduced. There are a few ideas about the API for preventing accidental
leaking [1]. Option 3 seems best from the POV of not needing to configure
anything in the normal case. I couldn't figure out how to do that, so guard it
with an experimental option for now.
[1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-January/092383.html
Anton Shestakov <av6@dwimlabs.net> [Tue, 17 Jan 2017 13:44:53 +0800] rev 30831
tests: test experimental.spacemovesdown config for commit -i
The feature is still very experimental, but at least its behavior is captured
in the test.
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 17 Jan 2017 10:17:13 -0800] rev 30830
zstd: prevent potential free() of uninitialized memory
This is a cherry pick of an upstream fix. The free() of uninitialed
memory could likely only occur if a malloc() inside zstd fails.
The patched functions aren't currently used by Mercurial. But I don't
like leaving footguns sitting around.
Martin von Zweigbergk <martinvonz@google.com> [Tue, 17 Jan 2017 11:25:02 -0800] rev 30829
revlog: give EXTSTORED flag value to narrowhg
Narrowhg has been using "1 << 14" as its revlog flag value for a long
time. We (Google) have many repos with that value in production
already. When the same value was reserved for EXTSTORED, it made those
repos invalid. Upgrading them will be a little painful. We should
clearly have reserved the value for narrowhg a long time ago. Since
the EXTSTORED flag is not yet in any release and Facebook also says
they have not started using it in production, so it should be okay to
change it. This patch gives the current value (1 << 14) back to
narrowhg and gives a new value (1 << 13) to EXTSTORED.
Martin von Zweigbergk <martinvonz@google.com> [Tue, 17 Jan 2017 11:45:10 -0800] rev 30828
help: don't let tools reflow revlog flags list
Before this change, the text about revlog flags was reflowed into a
single paragraph, which made it a bit hard to read. I don't even know
the rules around this, but adding a blank line before each flag seems
to prevent the reflowing.
Martin von Zweigbergk <martinvonz@google.com> [Tue, 17 Jan 2017 11:29:06 -0800] rev 30827
help: format revlog.txt more closely to result
The rendered text has spaces before each item in the list
Denis Laxalde <denis.laxalde@logilab.fr> [Tue, 17 Jan 2017 09:19:24 +0100] rev 30826
hgweb: simplify calculation of first revision in filelog command
Denis Laxalde <denis.laxalde@logilab.fr> [Tue, 17 Jan 2017 09:17:29 +0100] rev 30825
hgweb: restore ascending iteration on revs in filelog web command
Follow-up on 96f811bceb85. Adjust back the "parity" generator's offset to keep
rendering the same.
Denis Laxalde <denis.laxalde@logilab.fr> [Mon, 16 Jan 2017 09:22:32 +0100] rev 30824
context: extract _changesinrange() out of blockancestors()
We'll need it to write a blockdescendants function in next changeset.
Pulkit Goyal <7895pulkit@gmail.com> [Sat, 14 Jan 2017 01:23:07 +0530] rev 30823
shelve: allow multiple shelves with --patch and --stat
Before this patch, there was a single way to see multiple shelves using
`--patch --list` which show all the shelves. Doing `--patch s1 s2` returns an
error. This patch allows to show multiple shelves using `--patch` and `--stat`.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 14 Jan 2017 19:41:43 -0800] rev 30822
zstd: vendor python-zstandard 0.6.0
Commit 63c68d6f5fc8de4afd9bde81b13b537beb4e47e8 from
https://github.com/indygreg/python-zstandard is imported without
modifications (other than removing unwanted files).
This includes minor performance and feature improvements. It also
changes the vendored zstd library from 1.1.1 to 1.1.2.
# no-check-commit
Pulkit Goyal <7895pulkit@gmail.com> [Sat, 14 Jan 2017 20:05:15 +0530] rev 30821
util: add length argument to util.buffer()
util.buffer() either returns inbuilt buffer function or defines a new one which
slices. The inbuilt buffer() also has a length argument which is missing from
the ones we defined. This patch adds that length argument.
Pulkit Goyal <7895pulkit@gmail.com> [Sun, 15 Jan 2017 13:17:05 +0530] rev 30820
py3: replace pycompat.getenv with encoding.environ.get
pycompat.getenv returns os.getenvb on py3 which is not available on Windows.
This patch replaces them with encoding.environ.get and checks to ensure no
new instances of os.getenv or os.setenv are introduced.
Yuya Nishihara <yuya@tcha.org> [Sun, 15 Jan 2017 16:33:15 +0900] rev 30819
patch: check length of git index header only if integer is specified
Otherwise TypeError would be raised. Follows up d1901c4c8ec0.
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Jan 2017 20:16:56 -0800] rev 30818
localrepo: experimental support for non-zlib revlog compression
The final part of integrating the compression manager APIs into
revlog storage is the plumbing for repositories to advertise they
are using non-zlib storage and for revlogs to instantiate a non-zlib
compression engine.
The main intent of the compression manager work was to zstd all
of the things. Adding zstd to revlogs has proved to be more involved
than other places because revlogs are... special. Very small inputs
and the use of delta chains (which are themselves a form of
compression) are a completely different use case from streaming
compression, which bundles and the wire protocol employ. I've
conducted numerous experiments with zstd in revlogs and have yet
to formalize compression settings and a storage architecture that
I'm confident I won't regret later. In other words, I'm not yet
ready to commit to a new mechanism for using zstd - or any other
compression format - in revlogs.
That being said, having some support for zstd (and other compression
formats) in revlogs in core is beneficial. It can allow others to
conduct experiments.
This patch introduces *highly experimental* support for non-zlib
compression formats in revlogs. Introduced is a config option to
control which compression engine to use. Also introduced is a namespace
of "exp-compression-*" requirements to denote support for non-zlib
compression in revlogs. I've prefixed the namespace with "exp-"
(short for "experimental") because I'm not confident of the
requirements "schema" and in no way want to give the illusion of
supporting these requirements in the future. I fully intend to drop
support for these requirements once we figure out what we're doing
with zstd in revlogs.
A good portion of the patch is teaching the requirements system
about registered compression engines and passing the requested
compression engine as an opener option so revlogs can instantiate
the proper compression engine for new operations.
That's a verbose way of saying "we can now use zstd in revlogs!"
On an `hg pull` conversion of the mozilla-unified repo with no extra
redelta settings (like aggressivemergedeltas), we can see the impact
of zstd vs zlib in revlogs:
$ hg perfrevlogchunks -c
! chunk
! wall 2.032052 comb 2.040000 user 1.990000 sys 0.050000 (best of 5)
! wall 1.866360 comb 1.860000 user 1.820000 sys 0.040000 (best of 6)
! chunk batch
! wall 1.877261 comb 1.870000 user 1.860000 sys 0.010000 (best of 6)
! wall 1.705410 comb 1.710000 user 1.690000 sys 0.020000 (best of 6)
$ hg perfrevlogchunks -m
! chunk
! wall 2.721427 comb 2.720000 user 2.640000 sys 0.080000 (best of 4)
! wall 2.035076 comb 2.030000 user 1.950000 sys 0.080000 (best of 5)
! chunk batch
! wall 2.614561 comb 2.620000 user 2.580000 sys 0.040000 (best of 4)
! wall 1.910252 comb 1.910000 user 1.880000 sys 0.030000 (best of 6)
$ hg perfrevlog -c -d 1
! wall 4.812885 comb 4.820000 user 4.800000 sys 0.020000 (best of 3)
! wall 4.699621 comb 4.710000 user 4.700000 sys 0.010000 (best of 3)
$ hg perfrevlog -m -d 1000
! wall 34.252800 comb 34.250000 user 33.730000 sys 0.520000 (best of 3)
! wall 24.094999 comb 24.090000 user 23.320000 sys 0.770000 (best of 3)
Only modest wins for the changelog. But manifest reading is
significantly faster. What's going on?
One reason might be data volume. zstd decompresses faster. So given
more bytes, it will put more distance between it and zlib.
Another reason is size. In the current design, zstd revlogs are
*larger*:
debugcreatestreamclonebundle (size in bytes)
zlib: 1,638,852,492
zstd: 1,680,601,332
I haven't investigated this fully, but I reckon a significant cause of
larger revlogs is that the zstd frame/header has more bytes than
zlib's. For very small inputs or data that doesn't compress well, we'll
tend to store more uncompressed chunks than with zlib (because the
compressed size isn't smaller than original). This will make revlog
reading faster because it is doing less decompression.
Moving on to bundle performance:
$ hg bundle -a -t none-v2 (total CPU time)
zlib: 102.79s
zstd: 97.75s
So, marginal CPU decrease for reading all chunks in all revlogs
(this is somewhat disappointing).
$ hg bundle -a -t <engine>-v2 (total CPU time)
zlib: 191.59s
zstd: 115.36s
This last test effectively measures the difference between zlib->zlib
and zstd->zstd for revlogs to bundle. This is a rough approximation of
what a server does during `hg clone`.
There are some promising results for zstd. But not enough for me to
feel comfortable advertising it to users. We'll get there...
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Jan 2017 19:58:00 -0800] rev 30817
revlog: use compression engine APIs for decompression
Now that compression engines declare their header in revlog chunks
and can decompress revlog chunks, we refactor revlog.decompress()
to use them.
Making full use of the property that revlog compressor objects are
reusable, revlog instances now maintain a dict mapping an engine's
revlog header to a compressor object. This is not only a performance
optimization for engines where compressor object reuse can result in
better performance, but it also serves as a cache of header values
so we don't need to perform redundant lookups against the compression
engine manager. (Yes, I measured and the overhead of a function call
versus a dict lookup was observed.)
Replacing the previous inline lookup table with a dict lookup was
measured to make chunk reading ~2.5% slower on changelogs and ~4.5%
slower on manifests. So, the inline lookup table has been mostly
preserved so we don't lose performance. This is unfortunate. But
many decompression operations complete in microseconds, so Python
attribute lookup, dict lookup, and function calls do matter.
The impact of this change on mozilla-unified is as follows:
$ hg perfrevlogchunks -c
! chunk
! wall 1.953663 comb 1.950000 user 1.920000 sys 0.030000 (best of 6)
! wall 1.946000 comb 1.940000 user 1.910000 sys 0.030000 (best of 6)
! chunk batch
! wall 1.791075 comb 1.800000 user 1.760000 sys 0.040000 (best of 6)
! wall 1.785690 comb 1.770000 user 1.750000 sys 0.020000 (best of 6)
$ hg perfrevlogchunks -m
! chunk
! wall 2.587262 comb 2.580000 user 2.550000 sys 0.030000 (best of 4)
! wall 2.616330 comb 2.610000 user 2.560000 sys 0.050000 (best of 4)
! chunk batch
! wall 2.427092 comb 2.420000 user 2.400000 sys 0.020000 (best of 5)
! wall 2.462061 comb 2.460000 user 2.400000 sys 0.060000 (best of 4)
Changelog chunk reading is slightly faster but manifest reading is
slower. What gives?
On this repo, 99.85% of changelog entries are zlib compressed (the 'x'
header). On the manifest, 67.5% are zlib and 32.4% are '\0'. This patch
swapped the test order of 'x' and '\0' so now 'x' is tested first. This
makes changelogs faster since they almost always hit the first branch.
This makes a significant percentage of manifest '\0' chunks slower
because that code path now performs an extra test. Yes, I too can't
believe we're able to measure the impact of an if..elif with simple
string compares. I reckon this code would benefit from being written
in C...
Denis Laxalde <denis.laxalde@logilab.fr> [Fri, 13 Jan 2017 10:22:25 +0100] rev 30816
hgweb: build the "entries" list directly in filelog command
There's no apparent reason to have this "entries" generator function that
builds a list and then yields its elements in reverse order and which is only
called to build the "entries" list. So just build the list directly, in
reverse order.
Adjust "parity" generator's offset to keep rendering the same.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 14 Jan 2017 10:11:19 -0800] rev 30815
convert: remove "replacecommitter" action
As pointed out by Yuya, this action doesn't add much (any?) value.
Yuya Nishihara <yuya@tcha.org> [Sat, 14 Jan 2017 20:31:35 +0900] rev 30814
ui: check EOF of getpass() response read from command-server channel
readline() returns '' only when EOF is encountered, in which case, Python's
getpass() raises EOFError. We should do the same to abort the session as
"response expected."
This bug was reported to
https://bitbucket.org/tortoisehg/thg/issues/4659/
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Jan 2017 23:21:10 -0800] rev 30813
convert: config option to control Git committer actions
When converting a Git repository to Mercurial at Mozilla, I encountered
a scenario where I didn't want `hg convert` to automatically add the
"committer: <committer>" line to commit messages. While I can hack around
this by rewriting the Git commit before it is fed into `hg convert`,
I figured it would be a useful knob to control.
This patch introduces a config option that allows lots of control
over the committer value. I initially implemented this as a single
boolean flag to control whether to save the committer message. But
then there was feedback that it would be useful to save the committer
in extra data. While this patch doesn't implement support for saving
in extra data, it does add a mechanism for extending which actions
to take on the committer field. We should be able to easily add
actions to save in extra data.
Some of the implemented features weren't asked for. But I figured they
could be useful. If nothing else they demonstrate the extensibility
of this mechanism.
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Jan 2017 21:21:02 -0800] rev 30812
help: make "mergetool" an alias for "merge-tools"
I've probably typed `hg help mergetool` dozens of times. I'm tired
of it not working.
Matthieu Laneuville <mlaneuville@protonmail.com> [Thu, 12 Jan 2017 21:06:55 +0900] rev 30811
templatekw: force noprefix=False to insure diffstat consistency (issue4755)
The result of diffstatdata should not depend on having noprefix set or not, as
was reported in issue 4755. Forcing noprefix to false on call makes sure the
parser receives the diff in the correct format and returns the proper result.
Another way to fix this would have been to change the regular expressions in
path.diffstatdata(), but that would have introduced many unecessary special
cases.
Martin von Zweigbergk <martinvonz@google.com> [Fri, 13 Jan 2017 10:11:37 -0800] rev 30810
check-code: reject module-level @cachefunc
Module-level @cachefunc usage is risky because it can easily create a
memory "leak". Let's reject it completely for now. If a valid usage
comes up in the future, we can always improve the check or reconsider.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Fri, 13 Jan 2017 11:42:36 -0800] rev 30809
similar: remove caching from the module level
To prevent Bad Thingsâ„¢ from happening, let's rework the logic to not use
util.cachefunc.
Sean Farley <sean@farley.io> [Mon, 09 Jan 2017 11:01:45 -0800] rev 30808
patch: add label for coloring the similarity extended header
Just like the summary says, this will colorize the:
similarity index 88%
line in the diff output.
Sean Farley <sean@farley.io> [Mon, 09 Jan 2017 11:24:18 -0800] rev 30807
patch: use opt.showsimilarity to calculate and show the similarity
Tests have been added.
Sean Farley <sean@farley.io> [Mon, 09 Jan 2017 10:51:44 -0800] rev 30806
patch: add similarity config knob in experimental section
This config knob will control whether or not to show the similarity
calculation in the diff output:
diff --git a/README.md b/foo.md
similarity index 88%
rename from README.md
rename to foo.md
--- a/README.md
+++ b/foo.md
Sean Farley <sean@farley.io> [Sat, 07 Jan 2017 20:47:57 -0800] rev 30805
similar: move score function to module level
Future patches will use this to report the similarity of a rename / copy
in the patch output.
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Jan 2017 17:58:19 +0900] rev 30804
revset: abuse x:y syntax to specify line range of followlines()
This slightly complicates the parsing (see the previous patch), but the
overall result seems not bad.
I keep x:, :y and : for future extension.
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Jan 2017 16:55:56 +0900] rev 30803
revset: do not transform range* operators in parsed tree
This allows us to handle x:y range as a general range object. A primary user
of it is followlines().
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Jan 2017 17:45:11 +0900] rev 30802
revset: add default value to getinteger() helper
This seems handy.
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Jan 2017 17:39:44 +0900] rev 30801
revset: factor out getinteger() helper
We have 4 revset functions that take integer arguments, and they handle
their arguments in slightly different ways. This patch unifies them:
- getstring() in place of getsymbol(), which is more consistent with the
handling of integer revisions (both 1 and '1' are valid)
- say "expects" instead of "requires" for type errors
We don't need to catch TypeError since getstring() must return a string.
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Jan 2017 16:16:26 +0900] rev 30800
revset: rename rev argument of followlines() to startrev
The rev argument has the same meaning as startrev of follow(), and I think
startrev is more informative.
followlines() is new function, we can make BC now.
Yuya Nishihara <yuya@tcha.org> [Fri, 13 Jan 2017 23:48:21 +0900] rev 30799
help: use :hg: role and canonical name to point to revset string patterns
Follows up 5dd67f0993ce. Now revisions.txt and revsets.txt has been merged,
so use revisions.* as a pointer.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 02 Jan 2017 13:27:20 -0800] rev 30798
util: compression APIs to support revlog decompression
Previously, compression engines had APIs for performing revlog
compression but no mechanism to perform revlog decompression. This
patch changes that.
Revlog decompression is slightly more complicated than compression
because in the compression case there is (currently) only a single
engine that can be used at a time. However for decompression, a
revlog could contain chunks from multiple compression engines. This
means decompression needs to map to multiple engines and
decompressors. This functionality is outside the scope of this patch.
But it drives the decision for engines to declare a byte header
sequence that identifies revlog data as belonging to an engine and
an API for obtaining an engine from a revlog header.
Anton Shestakov <av6@dwimlabs.net> [Sun, 08 Jan 2017 10:08:29 +0800] rev 30797
crecord: add an experimental option for space key to move cursor down
I really want to have an option of toggling a selection on a line and also
moving cursor down as a single keystroke. It also kinda makes sense for space
key to do this, because some other curses UIs in the wild do this (e.g. various
file managers, htop). So I got an idea to make a config option that defaults to
False for compatibility, but allows making crecord UI a lot more useful for
people with big hunks.
We add this an experimental option to experiment with this behavior.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 02 Jan 2017 12:02:08 -0800] rev 30796
perf: support multiple compression engines in perfrevlogchunks
Now that the revlog has a reference to a compressor, it is
possible to swap in other compression engines. So, teach
`hg perfrevlogchunks` to do that.
The default behavior of `hg perfrevlogchunks` is now to measure the
compression performance of all compression engines implementing the
revlog compressor API. This effectively adds the no-op "none"
compressor and zstd (when available) into the default set.
While we can't yet plug alternate compressors into revlogs, this
command gives us a preview of the performance. On the mozilla-unified
repository:
$ hg perfrevlogchunks -c
! compress w/ none
! wall 0.115159 comb 0.110000 user 0.110000 sys 0.000000 (best of 86)
! compress w/ zlib
! wall 5.681406 comb 5.680000 user 5.680000 sys 0.000000 (best of 3)
! compress w/ zstd
! wall 2.624781 comb 2.620000 user 2.620000 sys 0.000000 (best of 4)
$ hg perfrevlogchunks -m
! compress w/ none
! wall 0.124486 comb 0.120000 user 0.120000 sys 0.000000 (best of 79)
! compress w/ zlib
! wall 10.144701 comb 10.150000 user 10.150000 sys 0.000000 (best of 3)
! compress w/ zstd
! wall 4.383118 comb 4.390000 user 4.390000 sys 0.000000 (best of 3)
Those numbers for zstd look promising. But they aren't the full story.
For that, we'll need to look at decompression times and storage sizes.
Stay tuned...
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 02 Jan 2017 11:22:52 -0800] rev 30795
revlog: use compression engine API for compression
This commit swaps in the just-added revlog compressor API into
the revlog class.
Instead of implementing zlib compression inline in compress(), we
now store a cached-on-first-use revlog compressor on each revlog
instance and invoke its "compress()" method.
As part of this, revlog.compress() has been refactored a bit to use
a cleaner code flow and modern formatting (e.g. avoiding
parenthesis around returned tuples).
On a mozilla-unified repo, here are the "compress" times for a few
commands:
$ hg perfrevlogchunks -c
! wall 5.772450 comb 5.780000 user 5.780000 sys 0.000000 (best of 3)
! wall 5.795158 comb 5.790000 user 5.790000 sys 0.000000 (best of 3)
$ hg perfrevlogchunks -m
! wall 9.975789 comb 9.970000 user 9.970000 sys 0.000000 (best of 3)
! wall 10.019505 comb 10.010000 user 10.010000 sys 0.000000 (best of 3)
Compression times did seem to slow down just a little. There are
360,210 changelog revisions and 359,342 manifest revisions. For the
changelog, mean time to compress a revision increased from ~16.025us to
~16.088us. That's basically a function call or an attribute lookup. I
suppose this is the price you pay for abstraction. It's so low that
I'm not concerned.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 02 Jan 2017 12:39:03 -0800] rev 30794
util: compression APIs to support revlog compression
As part of "zstd all of the things," we need to teach revlogs to
use non-zlib compression formats. Because we're routing all compression
via the "compression manager" and "compression engine" APIs, we need to
introduction functionality there for performing revlog operations.
Ideally, revlog compression and decompression operations would be
implemented in terms of simple "compress" and "decompress" primitives.
However, there are a few considerations that make us want to have a
specialized primitive for handling revlogs:
1) Performance. Revlogs tend to do compression and especially
decompression operations in batches. Any overhead for e.g.
instantiating a "context" for performing an operation can be
noticed. For this reason, our "revlog compressor" primitive is
reusable. For zstd, we reuse the same compression "context" for
multiple operations. I've measured this to have a performance
impact versus constructing new contexts for each operation.
2) Specialization. By having a primitive dedicated to revlog use,
we can make revlog-specific choices and leave the door open for
more functionality in the future. For example, the zstd revlog
compressor may one day make use of dictionary compression.
A future patch will introduce a decompress() on the compressor
object.
The code for the zlib compressor is basically copied from
revlog.compress(). Although it doesn't handle the empty input
case, the null first byte case, and the 'u' prefix case. These
cases will continue to be handled in revlog.py once that code is
ported to use this API.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 02 Jan 2017 13:00:16 -0800] rev 30793
revlog: move decompress() from module to revlog class (API)
Upcoming patches will convert revlogs to use the compression engine
APIs to perform all things compression. The yet-to-be-introduced
APIs support a persistent "compressor" object so the same object
can be reused for multiple compression operations, leading to
better performance. In addition, compression engines like zstd
may wish to tweak compression engine state based on the revlog
(e.g. per-revlog compression dictionaries).
A global and shared decompress() function will shortly no longer
make much sense. So, we move decompress() to be a method of the
revlog class. It joins compress() there.
On the mozilla-unified repo, we can measure the impact of this change
on reading performance:
$ hg perfrevlogchunks -c
! chunk
! wall 1.932573 comb 1.930000 user 1.900000 sys 0.030000 (best of 6)
! wall 1.955183 comb 1.960000 user 1.930000 sys 0.030000 (best of 6)
! chunk batch
! wall 1.787879 comb 1.780000 user 1.770000 sys 0.010000 (best of 6
! wall 1.774444 comb 1.770000 user 1.750000 sys 0.020000 (best of 6)
"chunk" appeared to become slower but "chunk batch" got faster. Upon
further examination by running both sets multiple times, the numbers
appear to converge across all runs. This tells me that there is no
perceived performance impact to this refactor.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 02 Jan 2017 11:50:17 -0800] rev 30792
revlog: make compressed size comparisons consistent
revlog.compress() compares the compressed size to the input size
and throws away the compressed data if it is larger than the input.
This is the correct thing to do, as storing compressed data that
is larger than the input takes up more storage space and makes reading
slower.
However, the comparison was implemented inconsistently. For the
streaming compression mode, we threw away the result if it was
greater than or equal to the input size. But for the one-shot
compression, we threw away the compression only if it was greater
than the input size!
This patch changes the comparison for the simple case so it is
consistent with the streaming case.
As a few tests demonstrate, this adds 1 byte to some revlog entries.
This is because of an added 'u' header on the chunk. It seems
somewhat wrong to increase the revlog size here. However, IMO the cost
of 1 byte in storage is insignificant compared to the performance gains
of avoiding decompression. This patch should invite questions around
the heuristic for throwing away compressed data. For example, I'd argue
we should be more liberal about rejecting compressed data, additionally
doing so where the number of bytes saved fails to reach a threshold.
But we can have this discussion another time.
Sean Farley <sean@farley.io> [Sat, 07 Jan 2017 20:43:49 -0800] rev 30791
similar: rename local variable to not collide with previous
Future patches will move the score function to the module level, so
let's not shadow that.
Sean Farley <sean@farley.io> [Mon, 09 Jan 2017 10:59:45 -0800] rev 30790
patch: add label for coloring the index extended header
Just like the summary says, this will colorize the:
index 3d3ba4b65e11..57274a0f46b2 100644
line in the diff output.
Sean Farley <sean@farley.io> [Sat, 31 Dec 2016 15:41:57 -0600] rev 30789
patch: add index line for diff output
This helps highlighting in third-party diff coloring (which assumes git
output) and maintains pedantic correctness with diff --git.
Tests will be added at the end of the series.
Sean Farley <sean@farley.io> [Mon, 09 Jan 2017 11:13:47 -0800] rev 30788
patch: add config knob for displaying the index header
This config knob can take an integer between 0 and 40 or a
keyword ('none', 'short', 'full') to control the length of hash to
output. It will display diffs with the git index header as such,
diff --git a/mercurial/mdiff.py b/mercurial/mdiff.py
index 112edf7..d6b52c5 100644
We'll put this in the experimental section for now.
Martin von Zweigbergk <martinvonz@google.com> [Thu, 12 Jan 2017 12:05:23 -0800] rev 30787
bisect: refer directly to bisect() revset predicate in help
We have specific syntax for displaying the help text for a particular
revset predicate, so let's refer directly to the bisect() revset in
the verbose bisect help. It seems likely that the user doesn't care
about other revsets at that point, so they will probably not miss the
text about the other revset predicates.
Martin von Zweigbergk <martinvonz@google.com> [Thu, 12 Jan 2017 11:43:26 -0800] rev 30786
tests: use "hg help revisions.<predicate>" instead of grepping
We have specific syntax for displaying the help text for a particular
revset predicate, so use that instead of grepping through the full
output.
Martin von Zweigbergk <martinvonz@google.com> [Thu, 12 Jan 2017 11:52:05 -0800] rev 30785
help: remove now-redundant pointer to revsets help
"hg help revisions" and "hg help revsets" now point to the same text,
so drop the revsets reference.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 07 Jan 2017 23:35:35 -0500] rev 30784
help: eliminate duplicate text for revset string patterns
There's no reason to duplicate this so many times, and it's likely an instance
will be missed if support for a new pattern is added and documented. The
stringmatcher is mostly used by revsets, though it is also used for the 'tag'
related templates, and namespace filtering in the journal extension. So maybe
there's a better place to document it. `hg help patterns` seems inappropriate,
because that is all file pattern matching.
While here, indicate how to perform case insensitive regex searches.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 07 Jan 2017 21:26:32 -0500] rev 30783
revset: add regular expression support to 'desc'
This is a case insensitive predicate like 'author', so it conforms to the
existing behavior of performing a case insensitive regex.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 11 Jan 2017 22:42:10 -0500] rev 30782
revset: stop lowercasing the regex pattern for 'author'
It was probably unintentional for regex, as the meaning of some sequences like
\S and \s is actually inverted by changing the case. For backward compatibility
however, the matching is forced to case insensitive.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 24 Nov 2016 18:45:29 -0800] rev 30781
repair: clean up stale lock file from store backup
Since we did a directory rename on the stores, the source
repository's lock path now references the dest repository's
lock path and the dest repository's lock path now references
a non-existent filename.
So releasing the lock on the source will unlock the dest and
releasing the lock on the dest will no-op because it fails due
to file not found. So we clean up the dest's lock manually.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 24 Nov 2016 18:34:50 -0800] rev 30780
repair: copy non-revlog store files during upgrade
The store contains more than just revlogs. This patch teaches the
upgrade code to copy regular files as well.
As the test changes demonstrate, the phaseroots file is now copied.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 18 Dec 2016 17:00:15 -0800] rev 30779
repair: migrate revlogs during upgrade
Our next step for in-place upgrade is to migrate store data. Revlogs
are the biggest source of data within the store and a store is useless
without them, so we implement their migration first.
Our strategy for migrating revlogs is to walk the store and call
`revlog.clone()` on each revlog. There are some minor complications.
Because revlogs have different storage options (e.g. changelog has
generaldelta and delta chains disabled), we need to obtain the
correct class of revlog so inserted data is encoded properly for its
type.
Various attempts at implementing progress indicators that didn't lead
to frustration from false "it's almost done" indicators were made.
I initially used a single progress bar based on number of revlogs.
However, this quickly churned through all filelogs, got to 99% then
effectively froze at 99.99% when it got to the manifest.
So I converted the progress bar to total revision count. This was a
little bit better. But the manifest was still significantly slower
than filelogs and it took forever to process the last few percent.
I then tried both revision/chunk bytes and raw bytes as the
denominator. This had the opposite effect: because so much data is in
manifests, it would churn through filelogs without showing much
progress. When it got to manifests, it would fill in 90+% of the
progress bar.
I finally gave up having a unified progress bar and instead implemented
3 progress bars: 1 for filelog revisions, 1 for manifest revisions, and
1 for changelog revisions. I added extra messages indicating the total
number of revisions of each so users know there are more progress bars
coming.
I also added extra messages before and after each stage to give extra
details about what is happening. Strictly speaking, this isn't
necessary. But the numbers are impressive. For example, when converting
a non-generaldelta mozilla-central repository, the messages you see are:
migrating 2475593 total revisions (1833043 in filelogs, 321156 in manifests, 321394 in changelog)
migrating 1.67 GB in store; 2508 GB tracked data
migrating 267868 filelogs containing 1833043 revisions (1.09 GB in store; 57.3 GB tracked data)
finished migrating 1833043 filelog revisions across 267868 filelogs; change in size: -415776 bytes
migrating 1 manifests containing 321156 revisions (518 MB in store; 2451 GB tracked data)
That "2508 GB" figure really blew me away. I had no clue that the raw
tracked data in mozilla-central was that large. Granted, 2451 GB is in
the manifest and "only" 57.3 GB is in filelogs. But still.
It's worth noting that gratuitous loading of source revlogs in order
to display numbers and progress bars does serve a purpose: it ensures
we can open all source revlogs. We don't want to spend several minutes
copying revlogs only to encounter a permissions error or similar later.
As part of this commit, we also add swapping of the store directory
to the upgrade function. After revlogs are converted, we move the
old store into the backup directory then move the temporary repo's
store into the old store's location. On well-behaved systems, this
should be 2 atomic operations and the window of inconsistency show be
very narrow.
There are still a few improvements to be made to store copying and
upgrading. But this commit gets the bulk of the work out of the way.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 18 Dec 2016 17:02:57 -0800] rev 30778
revlog: add clone method
Upcoming patches will introduce functionality for in-place
repository/store "upgrades." Copying the contents of a revlog
feels sufficiently low-level to warrant being in the revlog
class. So this commit implements that functionality.
Because full delta recomputation can be *very* expensive (we're
talking several hours on the Firefox repository), we support
multiple modes of execution with regards to delta (re)use. This
will allow repository upgrades to choose the "level" of
processing/optimization they wish to perform when converting
revlogs.
It's not obvious from this commit, but "addrevisioncb" will be
used for progress reporting.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 18 Dec 2016 16:59:04 -0800] rev 30777
repair: begin implementation of in-place upgrading
Now that all the upgrade planning work is in place, we can start
doing the real work: actually upgrading a repository.
The main goal of this commit is to get the "framework" for running
in-place upgrade actions in place.
Rather than get too clever and low-level with regards to in-place
upgrades, our strategy is to create a new, temporary repository,
copy data to it, then replace the old data with the new. This allows
us to reuse a lot of code in localrepo.py around store interaction,
which will eventually consume the bulk of the upgrade code.
But we have to start small. This patch implements adding new
repository requirements. But it still sets up a temporary
repository and locks it and the source repo before performing the
requirements file swap. This means all the plumbing is in place
to implement store copying in subsequent commits.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 18 Dec 2016 16:51:09 -0800] rev 30776
repair: determine what upgrade will do
This commit introduces code for determining what actions/improvements
an upgrade should perform.
The "upgradefindimprovements" function introduces a mechanism to
return a list of improvements that can be made to a repository.
Each improvement is effectively an action that an upgrade will
perform. Associated with each of these improvements is metadata
that will be used to inform users what's wrong and what an
upgrade will do.
Each "improvement" is categorized as a "deficiency" or an
"optimization." TBH, I'm not thrilled about the terminology and
am receptive to constructive bikeshedding. The main difference
between a "deficiency" and an "optimization" is a deficiency
is always corrected (if it deviates from the current config) and
an "optimization" is an optional action that goes above and beyond
to improve the state of the repository (usually by requiring more
CPU during upgrade).
Our initial set of improvements identifies missing repository
requirements, a single, easily correctable problem with
changelog storage, and a set of "optimizations" related to delta
recalculation.
The main "upgraderepo" function has been expanded to handle
improvements. It queries for the list of improvements and determines
which of them will run based on the current repository state and user
I went through numerous iterations of the output format before
settling on a ReST-inspired definition list format. (I used
bulleted lists in the first submission of this commit and could
not get it to format just right.) Even with the various iterations,
I'm still not super thrilled with the format. But, this is a debug*
command, so that should mean we can refine the output without BC
concerns.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 18 Dec 2016 16:16:54 -0800] rev 30775
repair: implement requirements checking for upgrades
This commit introduces functionality for upgrading a repository in
place. The first part that's implemented is testing for upgrade
"compatibility." This is done by examining repository requirements.
There are 5 functions returning sets of requirements that control
upgrading. Why so many functions? Mainly to support extensions.
Functions are easier to monkeypatch than module variables.
Astute readers will see that we don't support "manifestv2" and
"treemanifest" requirements in the upgrade mechanism. I don't have
a great answer for why other than this is a complex set of patches
and I don't want to deal with the complexity of these experimental
features just yet. We can teach the upgrade mechanism about them
later, once the basic upgrade mechanism is in place.
This commit also introduces the "upgraderepo" function. This will be
our main routine for performing an in-place upgrade. Currently, it
just implements requirements checking. The structure of some code in
this function may look a bit weird (e.g. the inline function that is
only called once). But this will make sense after future commits.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 24 Nov 2016 16:24:09 -0800] rev 30774
debugcommands: stub for debugupgraderepo command
Currently, if Mercurial introduces a new repository/store feature or
changes behavior of an existing feature, users must perform an
`hg clone` to create a new repository with hopefully the
correct/optimal settings. Unfortunately, even `hg clone` may not
give the correct results. For example, if you do a local `hg clone`,
you may get hardlinks to revlog files that inherit the old state.
If you `hg clone` from a remote or `hg clone --pull`, changegroup
application may bypass some optimization, such as converting to
generaldelta.
Optimizing a repository is harder than it seems and requires more
than a simple `hg` command invocation.
This commit starts the process of changing that. We introduce
`hg debugupgraderepo`, a command that performs an in-place upgrade
of a repository to use new, optimal features. The command is just
a stub right now. Features will be added in subsequent commits.
This commit does foreshadow some of the behavior of the new command,
notably that it doesn't do anything by default and that it takes
arguments that influence what actions it performs. These will be
explained more in subsequent commits.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 11 Jan 2017 21:47:19 -0500] rev 30773
util: teach stringmatcher to handle forced case insensitive matches
The 'author' and 'desc' revsets are documented to be case insensitive.
Unfortunately, this was implemented in 'author' by forcing the input to
lowercase, including for regex like '\B'. (This actually inverts the meaning of
the sequence.) For backward compatibility, we will keep that a case insensitive
regex, but by using matcher options instead of brute force.
This doesn't preclude future hypothetical 'icase-literal:' style prefixes that
can be provided by the user. Such user specified cases can probably be handled
up front by stripping 'icase-', setting the variable, and letting it drop
through the existing code.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 11 Jan 2017 23:13:51 -0500] rev 30772
revset: point to 'grep' in the 'keyword' help for regex searches
The help for 'grep' already points to 'keyword'.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 11 Jan 2017 23:13:00 -0800] rev 30771
help: explain that revsets can be used where 1 or 2 revs are wanted
We did not seem to document that one can do things like "hg up :@"
where the last revision of the revset ":@".
Martin von Zweigbergk <martinvonz@google.com> [Wed, 11 Jan 2017 22:46:07 -0800] rev 30770
help: explain what the term "revset" means
We refer to revsets in a few places (e.g. in "hg help config"), but we
never explained what they are. Until now.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 11 Jan 2017 11:37:38 -0800] rev 30769
help: merge revsets.txt into revisions.txt
Selecting single and multiple revisions is closely related, so let's
put it in one place, so users can easily find it. We actually did not
even point to "hg help revsets" from "hg help revisions", but now that
they're on a single page, that won't be necessary.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 11 Jan 2017 11:40:40 -0800] rev 30768
tests: use `hg help dates` instead of `hg help revs` in test
The revisions help is already long and will get longer, so switch to
another short and stable topic.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 11 Jan 2017 11:28:54 -0800] rev 30767
help: use a single paragraph to describe full and abbreviated nodeids
The texts describing 40-digit strings and the abbreviated form are
closely related, so make it a single paragraph.
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Jan 2017 23:37:08 -0800] rev 30766
hgweb: support Content Security Policy
Content-Security-Policy (CSP) is a web security feature that allows
servers to declare what loaded content is allowed to do. For example,
a policy can prevent loading of images, JavaScript, CSS, etc unless
the source of that content is whitelisted (by hostname, URI scheme,
hashes of content, etc). It's a nifty security feature that provides
extra mitigation against some attacks, notably XSS.
Mitigation against these attacks is important for Mercurial because
hgweb renders repository data, which is commonly untrusted. While we
make attempts to escape things, etc, there's the possibility that
malicious data could be injected into the site content. If this happens
today, the full power of the web browser is available to that
malicious content. A restrictive CSP policy (defined by the server
operator and sent in an HTTP header which is outside the control of
malicious content), could restrict browser capabilities and mitigate
security problems posed by malicious data.
CSP works by emitting an HTTP header declaring the policy that browsers
should apply. Ideally, this header would be emitted by a layer above
Mercurial (likely the HTTP server doing the WSGI "proxying"). This
works for some CSP policies, but not all.
For example, policies to allow inline JavaScript may require setting
a "nonce" attribute on <script>. This attribute value must be unique
and non-guessable. And, the value must be present in the HTTP header
and the HTML body. This means that coordinating the value between
Mercurial and another HTTP server could be difficult: it is much
easier to generate and emit the nonce in a central location.
This commit introduces support for emitting a
Content-Security-Policy header from hgweb. A config option defines
the header value. If present, the header is emitted. A special
"%nonce%" syntax in the value triggers generation of a nonce and
inclusion in <script> elements in templates. The inclusion of a
nonce does not occur unless "%nonce%" is present. This makes this
commit completely backwards compatible and the feature opt-in.
The nonce is a type 4 UUID, which is the flavor that is randomly
generated. It has 122 random bits, which should be plenty to satisfy
the guarantees of a nonce.
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Jan 2017 20:47:48 -0800] rev 30765
hgweb: call process_dates() via DOM event listener
All the hgweb templates include mercurial.js in their header. All
the hgweb templates have the same <script> boilerplate to run
process_dates(). This patch factors that function call into
mercurial.js as part of a DOMContentLoaded event listener.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Dec 2016 15:29:32 -0700] rev 30764
protocol: send application/mercurial-0.2 responses to capable clients
With this commit, the HTTP transport now parses the X-HgProto-<N>
header to determine what media type and compression engine to use for
responses. So far, we only compress responses that are already being
compressed with zlib today (stream response types to specific
commands). We can expand things to cover additional response types
later.
The practical side-effect of this commit is that non-zlib compression
engines will be used if both ends support them. This means if both
ends have zstd support, zstd - not zlib - will be used to compress
data!
When cloning the mozilla-unified repository between a local HTTP
server and client, the benefits of non-zlib compression are quite
noticeable:
engine server CPU (s) client CPU (s) bundle size
zlib (l=6) 174.1 283.2 1,148,547,026
zstd (l=1) 99.2 267.3 1,127,513,841
zstd (l=3) 103.1 266.9 1,018,861,363
zstd (l=7) 128.3 269.7 919,190,278
zstd (l=10) 162.0 - 894,547,179
none 95.3 277.2 4,097,566,064
The default zstd compression level is 3. So if you deploy zstd
capable Mercurial to your clients and servers and CPU time on
your server is dominated by "getbundle" requests (clients cloning
and pulling) - and my experience at Mozilla tells me this is often
the case - this commit could drastically reduce your server-side
CPU usage *and* save on bandwidth costs!
Another benefit of this change is that server operators can install
*any* compression engine. While it isn't enabled by default, the
"none" compression engine can now be used to disable wire protocol
compression completely. Previously, commands like "getbundle" always
zlib compressed output, adding considerable overhead to generating
responses. If you are on a high speed network and your server is under
high load, it might be advantageous to trade bandwidth for CPU.
Although, zstd at level 1 doesn't use that much CPU, so I'm not
convinced that disabling compression wholesale is worthwhile. And, my
data seems to indicate a slow down on the client without compression.
I suspect this is due to a lack of buffering resulting in an increase
in socket read() calls and/or the fact we're transferring an extra 3 GB
of data (parsing HTTP chunked transfer and processing extra TCP packets
can add up). This is definitely worth investigating and optimizing. But
since the "none" compressor isn't enabled by default, I'm inclined to
punt on this issue.
This commit introduces tons of tests. Some of these should arguably
have been implemented on previous commits. But it was difficult to
test without the server functionality in place.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Dec 2016 15:22:18 -0700] rev 30763
httppeer: advertise and support application/mercurial-0.2
Now that servers expose a capability indicating they support
application/mercurial-0.2 and compression, clients can key off
this to say they support responses that are compressed with
various compression formats.
After this commit, the HTTP wire protocol client now sends an
"X-HgProto-<N>" request header indicating its support for
"application/mercurial-0.2" media type and various compression
formats.
This commit also implements support for handling
"application/mercurial-0.2" responses. It simply reads the header
compression engine identifier then routes the remainder of the
response to the appropriate decompressor.
There were some test changes, but only to logging. That points to
an obvious gap in our test coverage. This will be addressed in a
subsequent commit once server support is in place (it is hard to
test without server support).
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Dec 2016 15:21:46 -0700] rev 30762
wireproto: advertise supported media types and compression formats
This commit introduces support for advertising a server's support for
media types and compression formats in accordance with the spec defined
in internals.wireproto.
The bulk of the new code is a helper function in wireproto.py to
obtain a prioritized list of compression engines available to the
wire protocol. While not utilized yet, we implement support
for obtaining the list of compression engines advertised by the
client.
The upcoming HTTP protocol enhancements are a bit lower-level than
existing tests (most existing tests are command centric). So,
this commit establishes a new test file that will be appropriate
for holding tests around the functionality of the HTTP protocol
itself.
Rounding out this change, `hg debuginstall` now prints compression
engines available to the server.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Dec 2016 13:51:12 -0700] rev 30761
util: declare wire protocol support of compression engines
This patch implements a new compression engine API allowing
compression engines to declare support for the wire protocol.
Support is declared by returning a compression format string
identifier that will be added to payloads to signal the compression
type of data that follows and default integer priorities of the
engine.
Accessor methods have been added to the compression engine manager
class to facilitate use.
Note that the "none" and "bz2" engines declare wire protocol support
but aren't enabled by default due to their priorities being 0. It
is essentially free from a coding perspective to support these
compression formats, so we do it in case anyone may derive use from
it.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Dec 2016 13:56:36 -0700] rev 30760
internals: document compression negotiation
As part of adding zstd support to all of the things, we'll need
to teach the wire protocol to support non-zlib compression formats.
This commit documents how we'll implement that.
To understand how we arrived at this proposal, let's look at how
things are done today.
The wire protocol today doesn't have a unified format. Instead,
there is a limited facility for differentiating replies as successful
or not. And, each command essentially defines its own response format.
A significant deficiency in the current protocol is the lack of
payload framing over the SSH transport. In the HTTP transport,
chunked transfer is used and the end of an HTTP response body (and
the end of a Mercurial command response) can be identified by a 0
length chunk. This is how HTTP chunked transfer works. But in the
SSH transport, there is no such framing, at least for certain
responses (notably the response to "getbundle" requests). Clients
can't simply read until end of stream because the socket is
persistent and reused for multiple requests. Clients need to know
when they've encountered the end of a request but there is nothing
simple for them to key off of to detect this. So what happens is
the client must decode the payload (as opposed to being dumb and
forwarding frames/packets). This means the payload itself needs
to support identifying end of stream. In some cases (bundle2), it
also means the payload can encode "error" or "interrupt" events
telling the client to e.g. abort processing. The lack of framing
on the SSH transport and the transfer of its responsibilities to
e.g. bundle2 is a massive layering violation and a wart on the
protocol architecture. It needs to be fixed someday by inventing a
proper framing protocol.
So about compression.
The client transport abstractions have a "_callcompressable()"
API. This API is called to invoke a remote command that will
send a compressible response. The response is essentially a
"streaming" response (no framing data at the Mercurial layer)
that is fed into a decompressor.
On the HTTP transport, the decompressor is zlib and only zlib.
There is currently no mechanism for the client to specify an
alternate compression format. And, clients don't advertise what
compression formats they support or ask the server to send a
specific compression format. Instead, it is assumed that non-error
responses to "compressible" commands are zlib compressed.
On the SSH transport, there is no compression at the Mercurial
protocol layer. Instead, compression must be handled by SSH
itself (e.g. `ssh -C`) or within the payload data (e.g. bundle
compression).
For the HTTP transport, adding new compression formats is pretty
straightforward. Once you know what decompressor to use, you can
stream data into the decompressor until you reach a 0 size HTTP
chunk, at which point you are at end of stream.
So our wire protocol changes for the HTTP transport are pretty
straightforward: the client and server advertise what compression
formats they support and an appropriate compression format is
chosen. We introduce a new HTTP media type to hold compressed
payloads. The header of the payload defines the compression format
being used. Whoever is on the receiving end can sniff the first few
bytes route to an appropriate decompressor.
Support for multiple compression formats is advertised on both
server and client. The server advertises a "compression" capability
saying which compression formats it supports and in what order they
are preferred. Clients advertise their support for multiple
compression formats and media types via the introduced "X-HgProto"
request header.
Strictly speaking, servers don't need to advertise which compression
formats they support. But doing so allows clients to fail fast if
they don't support any of the formats the server does. This is useful
in situations like sending bundles, where the client may have to
perform expensive computation before sending data to the server.
Rather than simply advertise a list of supported compression formats,
we introduce an additional "httpmediatype" server capability
advertising which media types the server supports. This means servers
are explicit about what formats they exchange. IMO, this is superior
to inferring support from other capabilities (like "compression").
By advertising compression support on each request in the "X-HgProto"
header and media type and direction at the server level, we are able
to gradually transition existing commands/responses to the new media
type and possibly compression. Contrast with the old world, where we
only supported a single media type and the use of compression was
built-in to the semantics of the command on both client and server.
In the new world, if "application/mercurial-0.2" is supported,
compression is supported. It's that simple.
It's worth noting that we explicitly don't use "Accept,"
"Accept-Encoding," "Content-Encoding," or "Transfer-Encoding" for
content negotiation and compression. People knowledgeable of the HTTP
specifications will say that we should use these because that's
what they are designed to be used for. They have a point and I
sympathize with the argument. Earlier versions of this commit even
defined supported media types in the "Accept" header. However, my
years of experience rolling out services leveraging HTTP has taught
me to not trust the HTTP layer, especially if you are going outside
the normal spec (such as using a custom "Content-Encoding" value to
represent zstd streams). I've seen load balancers, proxies, and other
network devices do very bad and unexpected things to HTTP messages
(like insisting zlib compressed content is decoded and then re-encoded
at a different compression level or even stripping compression
completely). I've found that the best way to avoid surprises when
writing protocols on top of HTTP is to use HTTP as a dumb transport as
much as possible to minimize the chances that an "intelligent" agent
between endpoints will muck with your data. While the widespread use of
TLS is mitigating many intermediate network agents interfering with
HTTP, there are still problems at the edges, with e.g. the origin HTTP
server needing to convert HTTP to and from WSGI and buggy or
feature-lacking HTTP client implementations. I've found the best way to
avoid these problems is to avoid using headers like "Content-Encoding"
and to bake as much logic as possible into media types and HTTP message
bodies. The protocol changes in this commit do rely on a custom HTTP
request header and the "Content-Type" headers. But we used them before,
so we shouldn't be increasing our exposure to "bad" HTTP agents.
For the SSH transport, we can't easily implement content negotiation
to determine compression formats because the SSH transport has no
content negotiation capabilities today. And without a framing protocol,
we don't know how much data to feed into a decompressor. So in order
to implement compression support on the SSH transport, we'd need to
invent a mechanism to represent content types and an outer framing
protocol to stream data robustly. While I'm fully capable of doing
that, it is a lot of work and not something that should be undertaken
lightly. My opinion is that if we're going to change the SSH transport
protocol, we should take a long hard look at implementing a grand
unified protocol that attempts to address all the deficiencies with
the existing protocol. While I want this to happen, that would be
massive scope bloat standing in the way of zstd support. So, I've
decided to take the easy solution: the SSH transport will not gain
support for multiple compression formats. Keep in mind it doesn't
support *any* compression today. So essentially nothing is changing
on the SSH front.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Dec 2016 14:46:02 -0700] rev 30759
httppeer: extract code for HTTP header spanning
A second consumer of HTTP header spanning will soon be introduced.
Factor out the code to do this so it can be reused.
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Jan 2017 11:20:32 -0800] rev 30758
commands: config option to control bundle compression level
Currently, bundle compression uses the default compression level
for the active compression engine. The default compression level
is tuned as a compromise between speed and size.
Some scenarios may call for a different compression level. For
example, with clone bundles, bundles are generated once and used
several times. Since the cost to generate is paid infrequently,
server operators may wish to trade extra CPU time for better
compression ratios.
This patch introduces an experimental and undocumented config
option to control the bundle compression level. As the inline
comment says, this approach is a bit hacky. I'd prefer for
the compression level to be encoded in the bundle spec. e.g.
"zstd-v2;complevel=15." However, given that the 4.1 freeze is
imminent, I'm not comfortable implementing this user-facing
change without much time to test and consider the implications.
So, we're going with the quick and dirty solution for now.
Having this option in the 4.1 release will enable Mozilla to
easily produce and test zlib and zstd bundles with non-default
compression levels in production. This will help drive future
development of the feature and zstd integration with Mercurial.
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Jan 2017 11:19:37 -0800] rev 30757
bundle2: allow compression options to be passed to compressor
Compression engines allow options to be passed to them to control
behavior. This patch exposes an argument to bundle2.writebundle()
that passes options to the compression engine when writing compressed
bundles. The argument is honored for both bundle1 and bundle2, the
latter requiring a bit of plumbing to pass the value around.
Jun Wu <quark@fb.com> [Wed, 11 Jan 2017 23:39:24 +0800] rev 30756
chg: check snprintf result strictly
This makes the program more robust when somebody changes hgclient's
maxdatasize in the future.
Valters Vingolds <valters@vingolds.ch> [Tue, 10 Jan 2017 09:32:27 +0100] rev 30755
rebase: provide detailed hint to abort message if working dir is not clean
Detailed hint message is now provided when 'pull --rebase' operation detects
unclean working dir, for example:
abort: uncommitted changes
(cannot pull with rebase: please commit or shelve your changes first)
Added tests for uncommitted merge, and for subrepo support verifying that same
hint is also passed to subrepo state check.
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Jan 2017 16:02:56 +0900] rev 30754
revset: parse variable-length arguments of followlines() by getargsdict()
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Jan 2017 15:25:52 +0900] rev 30753
parser: extend buildargsdict() to support variable-length positional args
This can simplify the argument parsing of followlines(). Tests are added by
the next patch.
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Jan 2017 15:15:21 +0900] rev 30752
parser: make buildargsdict() precompute position where keyword args start
This prepares for adding *varargs support. See the next patch.
Jun Wu <quark@fb.com> [Wed, 11 Jan 2017 07:40:52 +0800] rev 30751
chg: change server's process title
This patch uses the newly introduced "setprocname" interface to update the
process title server-side, to make it easier to tell what a worker is actually
doing.
The new title is "chg[worker/$PID]", where PID is the process ID of the
connected client. It can be directly observed using "ps -AF" under Linux, or
"ps -A" under FreeBSD.
Jun Wu <quark@fb.com> [Wed, 11 Jan 2017 07:36:48 +0800] rev 30750
chgserver: add the setprocname interface
This allows clients to change its process title freely.
Anton Shestakov <av6@dwimlabs.net> [Tue, 10 Jan 2017 23:41:58 +0800] rev 30749
hgweb: use archivespecs for links on repo index page too
Moving archivespecs to the module level allows using it from other modules
(such as hgwebdir_mod), and keeping a reference to it in requestcontext allows
current code to just work.
Anton Shestakov <av6@dwimlabs.net> [Tue, 10 Jan 2017 23:34:39 +0800] rev 30748
hgweb: use util.sortdict for archivespecs
Thus we allow dict-like indexing and "in" checks, and also preserve the order
of archive types and can generate links in a certain order (so
requestcontext.archives is no longer needed).
Anton Shestakov <av6@dwimlabs.net> [Wed, 11 Jan 2017 01:25:07 +0800] rev 30747
hgweb: test the order of archive links
Remi Chaintron <remi@fb.com> [Thu, 05 Jan 2017 17:16:51 +0000] rev 30746
revlog: REVIDX_EXTSTORED flag
This flag will be used by the lfs extension to mark the revision data as stored
externally.
Remi Chaintron <remi@fb.com> [Tue, 10 Jan 2017 16:15:21 +0000] rev 30745
revlog: flag processor
Add the ability for revlog objects to process revision flags and apply
registered transforms on read/write operations.
This patch introduces:
- the 'revlog._processflags()' method that looks at revision flags and applies
flag processors registered on them. Due to the need to handle non-commutative
operations, flag transforms are applied in stable order but the order in which
the transforms are applied is reversed between read and write operations.
- the 'addflagprocessor()' method allowing to register processors on flags.
Flag processors are defined as a 3-tuple of (read, write, raw) functions to be
applied depending on the operation being performed.
- an update on 'revlog.addrevision()' behavior. The current flagprocessor design
relies on extensions to wrap around 'addrevision()' to set flags on revision
data, and on the flagprocessor to perform the actual transformation of its
contents. In the lfs case, this means we need to process flags before we meet
the 2GB size check, leading to performing some operations before it happens:
- if flags are set on the revision data, we assume some extensions might be
modifying the contents using the flag processor next, and we compute the
node for the original revision data (still allowing extension to override
the node by wrapping around 'addrevision()').
- we then invoke the flag processor to apply registered transforms (in lfs's
case, drastically reducing the size of large blobs).
- finally, we proceed with the 2GB size check.
Note: In the case a cachedelta is passed to 'addrevision()' and we detect the
flag processor modified the revision data, we chose to trust the flag processor
and drop the cachedelta.
Remi Chaintron <remi@fb.com> [Thu, 05 Jan 2017 17:16:07 +0000] rev 30744
revlog: pass revlog flags to addrevision
Adding the ability to passing flags to addrevision instead of simply passing
default flags to _addrevision will allow extensions relying on flag transforms
to wrap around addrevision() in order to update revlog flags.
The first use case of this patch will be the lfs extension marking nodes as
stored externally when the contents are larger than the defined threshold.
One of the reasons leading to setting flags in addrevision() wrappers in the
flag processor design is that it allows to detect files larger than the 2GB
limit before the check is performed, which allows lfs to transform the contents
into metadata.
Remi Chaintron <remi@fb.com> [Thu, 05 Jan 2017 17:16:07 +0000] rev 30743
revlog: add 'raw' argument to revision and _addrevision
This patch introduces a new 'raw' argument (defaults to False) to revlog's
revision() and _addrevision() methods.
When the 'raw' argument is set to True, it indicates the revision data should be
handled as raw data by the flagprocessor.
Note: Given revlog.addgroup() calls are restricted to changegroup generation, we
can always set raw to True when calling revlog._addrevision() from
revlog.addgroup().
Jun Wu <quark@fb.com> [Tue, 10 Jan 2017 06:59:49 +0800] rev 30742
pager: do not special case chg
Since chg has its own _runpager implementation, it's no longer necessary to
special-case chg in the pager extension. This will effectively enable the
new chg pager code path that runs inside runcommand.
Jun Wu <quark@fb.com> [Tue, 10 Jan 2017 06:59:39 +0800] rev 30741
chg: remove getpager support
We have enough bits to switch to the new chg pager code path in runcommand.
So just remove the legacy getpager support.
This is a red-only patch, and will break chg's pager support temporarily.
Jun Wu <quark@fb.com> [Tue, 10 Jan 2017 06:59:31 +0800] rev 30740
chgserver: implement chgui._runpager
This patch implements chgui._runpager in a relatively simple way. A more
clean way is to move the core logic of "attachio" to "ui", which will be
done later after chg runs uisetup per request.
Jun Wu <quark@fb.com> [Tue, 10 Jan 2017 06:59:21 +0800] rev 30739
chgserver: make S channel support pager request
This patch adds the "pager" support for the S channel. The pager API allows
running some subcommands, namely attachio, and waiting for the client to be
properly synchronized.
Jun Wu <quark@fb.com> [Tue, 10 Jan 2017 06:59:03 +0800] rev 30738
chg: handle pager request client-side
This patch implements the simple S-channel pager handling at chg
client-side.
Note: It does not deal with environ and cwd currently for simplicity, which
will be fixed later.
Jun Wu <quark@fb.com> [Tue, 10 Jan 2017 06:58:51 +0800] rev 30737
chgserver: use util.shellenviron
This avoids code duplication.
Jun Wu <quark@fb.com> [Tue, 10 Jan 2017 06:58:02 +0800] rev 30736
util: extract the logic calculating environment variables
The method will be reused in chgserver. Move it out so it can be reused.
Anton Shestakov <av6@dwimlabs.net> [Sun, 08 Jan 2017 00:52:54 +0800] rev 30735
hgweb: generate archive links in order
It would be nice for archive links to always be in a certain commonly used
order, such as 'zip', 'bz', 'gzip2'. Repo index page (hgwebdir_mod) already
shows archive links in this order, let's do the same in hgweb_mod.
Sadly, archivespecs is a regular unordered dict, and collections.OrderedDict is
new in 2.7. But requestcontext.archives is a tuple of archive types, so it can
be used as an index to archivespecs.
Anton Shestakov <av6@dwimlabs.net> [Sun, 08 Jan 2017 01:24:45 +0800] rev 30734
hgweb: use archivespecs (dict) instead of archives (tuple) for "in" check
Matt Harbison <matt_harbison@yahoo.com> [Sun, 08 Jan 2017 14:37:44 -0500] rev 30733
test-obsolete: stabilize output on platforms without 'serve' support
The conditional was updating the repository, which wasn't reflected in
subsequent logs on Windows, so the conditional is narrowed to just the serve
commands. The serve operation generates log files, so those are deleted to keep
the output of summary consistent.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 08 Jan 2017 13:49:53 -0500] rev 30732
tests: update globs for Windows
The extra glob in test-command-template.t caused it to say no result was
reported. It used to be (within the past year), that both this and the missing
glob cases could be fixed simply by editing any output in the test, and
re-running it in interactive mode. But that no longer works, and I had to diff
*.t against *.t.err. I didn't dig into what changed.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 08 Jan 2017 12:05:10 -0500] rev 30731
help: merge the various operator sections of revsets, filesets and templates
Having sections for specific operator types assumes the user already knows what
type of operators are supported. By having a common heading, the user can
simply lookup help for "(revsets|filesets|templates).operators".
Matt Harbison <matt_harbison@yahoo.com> [Sun, 08 Jan 2017 02:43:01 -0500] rev 30730
help: apply the section headings from revsets to templates
Unlike filesets, there are a few distinct headings that are not shared with
revsets. But common names are used where possible.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 08 Jan 2017 02:40:36 -0500] rev 30729
help: apply the section headings from revsets to filesets
This has the nice property of visually breaking up the wall of text. It also
allows specific smaller sections to be called out. For example,
`hg help filesets.predicates` now prints just the predicate section. At the
moment, the revset headings are a superset of the fileset headings, so there is
consistency in how example, predicate and operator help is called out.
The reference to `hg help patterns` was moved to the overview section, so that
it isn't stuck in the examples section.
Jun Wu <quark@fb.com> [Fri, 06 Jan 2017 16:14:52 +0000] rev 30728
chg: check type read from S channel
The previous patch added the check server-side. This patch added it
client-side.
Jun Wu <quark@fb.com> [Fri, 06 Jan 2017 16:12:25 +0000] rev 30727
chgserver: check type passed to S channel
It currently only supports the "system" type. Add an explicit check.
Jun Wu <quark@fb.com> [Fri, 06 Jan 2017 16:11:03 +0000] rev 30726
chg: send type information via S channel (BC)
Previously S channel is only used to send system commands. It will also be
used to send pager commands. So add a type parameter.
This breaks older chg clients. But chg and hg should always come from a
single commit and be packed into a single package. Supporting running
inconsistent versions of chg and hg seems to be unnecessarily complicated
with little benefit. So just make the change and assume people won't use
inconsistent chg with hg.
Valters Vingolds <valters@vingolds.ch> [Sun, 01 Jan 2017 13:16:29 +0100] rev 30725
rebase: fail-fast the pull if working dir is not clean (BC)
Refuse to run 'hg pull --rebase' if there are uncommitted changes:
so that instead of going ahead with fetching changes and then suddenly aborting
the rebase, we can warn user of uncommitted changes (or unclean repo state)
right up front.
In tests, we create a 'histedit' session to verify that also an unfinished
state is detected and handled.
Yuya Nishihara <yuya@tcha.org> [Fri, 06 Jan 2017 22:50:04 +0900] rev 30724
commit: fix unmodified message detection for the "--- >8 ----" magic
We need the raw editortext to be compared with the templatetext.
Yuya Nishihara <yuya@tcha.org> [Fri, 06 Jan 2017 22:44:39 +0900] rev 30723
commit: update test to actually modify template text
We have a check for unmodified commit message (introduced by bec1a579ebc4),
which should be enabled for the "--- >8 ---" magic but currently not.
Jun Wu <quark@fb.com> [Mon, 26 Dec 2016 00:25:44 +0000] rev 30722
pager: wrap ui._runpager
As discussed at [1], ui._runpager will be the new low-level API accepting a
pager command to actually run the pager. And ui.pager is the high-level API
which reads config directly from self.
This change is necessary for chgserver to override _runpager cleanly.
[1]: www.mercurial-scm.org/pipermail/mercurial-devel/2016-December/091656.html
Denis Laxalde <denis@laxalde.org> [Sat, 07 Jan 2017 12:24:15 +0100] rev 30721
summary: use ui.label and join to write evolution troubles
Follow-up on 7b526670f540 to avoid a convoluted loop.
Denis Laxalde <denis@laxalde.org> [Sat, 07 Jan 2017 12:07:56 +0100] rev 30720
log: drop unnecessary ui.note label from "trouble: " line
Follow-up on f05ede08dcf7 and 6d0b1a69f98c.
Denis Laxalde <denis.laxalde@logilab.fr> [Wed, 04 Jan 2017 16:47:49 +0100] rev 30719
revset: add a followlines(file, fromline, toline[, rev]) revset
This revset returns the history of a range of lines (fromline, toline) of a
file starting from `rev` or the current working directory.
Added tests in test-annotate.t which already contains a reasonably complex
repository.
Denis Laxalde <denis.laxalde@logilab.fr> [Wed, 28 Dec 2016 23:03:37 +0100] rev 30718
context: add a `blockancestors(fctx, fromline, toline)` function
This yields ancestors of `fctx` by only keeping changesets touching the file
within specified linerange = (fromline, toline).
Matching revisions are found by inspecting the result of `mdiff.allblocks()`,
filtered by `mdiff.blocksinrange()`, to find out if there are blocks of type
"!" within specified line range.
If, at some iteration, an ancestor with an empty line range is encountered,
the algorithm stops as it means that the considered block of lines actually
has been introduced in the revision of this iteration. Otherwise, we finally
yield the initial revision of the file as the block originates from it.
When a merge changeset is encountered during ancestors lookup, we consider
there's a diff in the current line range as long as there is a diff between
the merge changeset and at least one of its parents (in the current line
range).
Denis Laxalde <denis.laxalde@logilab.fr> [Tue, 03 Jan 2017 18:15:58 +0100] rev 30717
mdiff: add a "blocksinrange" function to filter diff blocks by line range
The function filters diff blocks as generated by mdiff.allblock function based
on whether they are contained in a given line range based on the "b-side" of
blocks.
Jun Wu <quark@fb.com> [Fri, 06 Jan 2017 16:19:41 +0000] rev 30716
run-tests: unset CHGDEBUG
With CHGDEBUG, chg outputs much more stuff and the test could fail running
with --chg. So unset the environment variable.
Denis Laxalde <denis.laxalde@logilab.fr> [Fri, 06 Jan 2017 14:35:22 +0100] rev 30715
summary: add evolution "troubles" information to summary output
Extend the "parent: " lines in summary with the list of evolution "troubles"
in parentheses, when the parent is troubled.
Denis Laxalde <denis.laxalde@logilab.fr> [Fri, 06 Jan 2017 14:34:34 +0100] rev 30714
summary: use the same labels as log command in "parent: " line
Re-use the cmdutil._changesetlabels function introduced in 5289fd78017a to
have consistent labels between the "changeset: " line in log command and the
"parent: " line in summary.
Denis Laxalde <denis.laxalde@logilab.fr> [Fri, 06 Jan 2017 13:50:52 +0100] rev 30713
templates: display evolution "troubles" in command line style
Denis Laxalde <denis.laxalde@logilab.fr> [Fri, 06 Jan 2017 13:50:16 +0100] rev 30712
templatekw: add a "troubles" template keyword
The "troubles" template keyword returns a list of evolution troubles.
It is EXPERIMENTAL, as anything else related to changeset evolution.
Test it in test-obsolete.t which has troubled changesets.
Denis Laxalde <denis.laxalde@logilab.fr> [Fri, 06 Jan 2017 15:48:22 +0100] rev 30711
test: test "trouble: " line in log output with multiple troubles
Follow-up on f05ede08dcf7.
Denis Laxalde <denis.laxalde@logilab.fr> [Fri, 06 Jan 2017 12:36:21 +0100] rev 30710
cmdutil: add missing "i18n" comment about "trouble: " line
Follow-up on f05ede08dcf7 per late review.
Valters Vingolds <valters@vingolds.ch> [Fri, 06 Jan 2017 09:58:41 +0100] rev 30709
rebase: use repo.vfs.exists in 'hg summary' hook
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 28 Dec 2016 15:48:17 -0700] rev 30708
hgweb: link to raw-file on annotation page (BC)
Every other template has the "raw" link load "raw-file." However,
fileannotate.tmpl's "raw" link loads "raw-annotate." This feels
inconsistent and wrong.
As far as I can tell, linking to the "raw annotate" view has occurred
since 2006.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 04 Jan 2017 10:35:04 -0800] rev 30707
repair: combine two loops over changelog revisions
This just saves a few lines.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 04 Jan 2017 10:07:12 -0800] rev 30706
repair: speed up stripping of many roots
repair.strip() expects a set of root revisions to strip. It then
builds the full set of descedants by walking the descandants of
each. It is rare that more than a few roots get passed in, but if that
happens, it will wastefully walk the changelog for each root. So let's
just walk it once.
I noticed this because the narrowhg extension was passing not only
roots, but all the commits to strip. When there were tens of thousands
of commits to strip, this resulted in quadratic behavior with that
extension.
Anton Shestakov <av6@dwimlabs.net> [Fri, 06 Jan 2017 09:56:40 +0800] rev 30705
make: remove targets for building packages for ubuntu wily (end of life)
Ubuntu 15.10 (Wily Werewolf) came out on October 22, 2015 and reached end of
life on July 28, 2016 [1]. Users were encouraged to upgrade to 16.04 (Xenial).
PPA doesn't allow new uploads targeting 15.10 anymore.
[1]: https://wiki.ubuntu.com/Releases
Sean Farley <sean@farley.io> [Wed, 04 Jan 2017 22:32:42 -0600] rev 30704
config: add docs for ignoring all text below in the editor
This is an example of how to use the new skip-from-there string for ignoring the
diff in a commit message.
Sean Farley <sean@farley.io> [Sat, 31 Dec 2016 15:36:36 -0600] rev 30703
cmdutil: add special string that ignores rest of text
Similar to git, we add a special string:
HG: ------------------------ >8 ------------------------
that means anything below it is ignored in a commit message.
This is helpful for integrating with third-party tools that display the
Yuya Nishihara <yuya@tcha.org> [Sat, 14 May 2016 20:52:44 +0900] rev 30702
revset: drop TODO comment about sorting issue of fullreposet
The bootstrapping issue was addressed at the parsing phase and we expect
that fullreposet.__and__() fully complies to the smartset API, in which
'self & other' should return a result set in self's order. See also
90455e7bf543.
Yuya Nishihara <yuya@tcha.org> [Thu, 05 Jan 2017 22:53:42 +0900] rev 30701
revset: document wdir() as an experimental function
Let's resurrect the docstring since our help module can detect the EXPERIMENTAL
tag and display it only if -v is specified.
This patch updates the test added by 015c0d1087a3 since wdir() is now
documented.
Yuya Nishihara <yuya@tcha.org> [Sat, 20 Aug 2016 17:50:23 +0900] rev 30700
revset: categorize wdir() as very fast function
The cost of wdir() should be identical to or cheaper than _intlist().
Yuya Nishihara <yuya@tcha.org> [Sat, 23 May 2015 11:04:11 +0900] rev 30699
revset: make children() not look at p2 if null (issue5439)
Unlike p1 = null, p2 = null denotes the revision has only one parent, which
shouldn't be considered a child of the null revision. This was spotted while
fixing the issue4682 and rediscovered as issue5439.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Jan 2017 19:17:44 -0800] rev 30698
setup: pass named argument to setup_zstd
The next release from upstream adds another named argument to this
function. Specify arguments by name so there is no ambiguity about
which argument is being passed.
Augie Fackler <augie@google.com> [Wed, 04 Jan 2017 14:52:59 -0500] rev 30697
merge with stable
Denis Laxalde <denis.laxalde@logilab.fr> [Tue, 03 Jan 2017 13:25:29 +0100] rev 30696
templates-default: factor out definition of changeset labels
This is redundant for normal and debug mode and prepares extension of this
list that should effect both modes.
Denis Laxalde <denis.laxalde@logilab.fr> [Mon, 10 Oct 2016 12:06:58 +0200] rev 30695
cmdutil: add support for evolution "troubles" display in changeset_printer
Add a "trouble" line in changeset header along with a couple of labels on
"log.changeset" line to indicate whether a changeset is troubled or not and
which kind trouble occurs.
Denis Laxalde <denis.laxalde@logilab.fr> [Tue, 03 Jan 2017 10:56:41 +0100] rev 30694
cmdutil: extract a _changesetlabels function out of changeset_printer._show()
There is a common logic in changeset_printer and in the summary command for
labelling a changeset.
This prepares extension of changeset's labels with evolution "troubles"
information that would show up in both log and summary outputs. Ultimately,
both would use this function.
Jun Wu <quark@fb.com> [Mon, 02 Jan 2017 14:57:14 +0000] rev 30693
chg: add procutil.h
This patch adds a formal header procutil.h for procutil.c, and changes
Makefile to build procutil.c independently.
Jun Wu <quark@fb.com> [Mon, 02 Jan 2017 14:43:37 +0000] rev 30692
chg: let procutil maintain its own pagerpid
Previously, chg.c maintains the pagerpid. Let's move it to procutil.c.
Note: chg.c still have a pagerpid to decide whether to call attachio or not.
In the future, attachio may be moved from hgc_open to hgc_runcommand, and
hgc_runcommand handles both pager and attachio so we don't need to run
attachio twice. And chg.c will be free of pagerpid.
Jun Wu <quark@fb.com> [Mon, 02 Jan 2017 14:10:32 +0000] rev 30691
chg: decouple hgclient from setuppager
procutil should not depend on hgclient. This patch makes the pager handling
part independent from hgclient.
Jun Wu <quark@fb.com> [Mon, 02 Jan 2017 14:04:35 +0000] rev 30690
chg: decouple hgclient from setupsignalhandler
procutil should not depend on hgclient. This patch makes the signal handling
part independent from hgclient.
Jun Wu <quark@fb.com> [Mon, 02 Jan 2017 14:02:47 +0000] rev 30689
chg: move signal and pager handling to a separate file
In the future hgclient will deal with pager directly inside runcommand, so
related signal handling stuff needs to be decoupled from chg.c.
The signal handling and pager logic are coupled because we need to forward
SIGPIPE when pager exits. So they are moved together, otherwise a global
variable (pagerpid) is inevitable.
This patch moves related functions from chg.c to procutil.c, which was
marked as copied to maintain annotate history.
The move is done without code modification for easy review, therefore
`#include "procutil.c"` was introduced temporarily.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 07 Oct 2015 16:02:45 -0700] rev 30688
keepalive: rewrite readline()
The old method was performing string concatenation, which is slower than
collecting raw chunks in a list and joining at the end.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 07 Oct 2015 15:53:58 -0700] rev 30687
keepalive: remove limit argument from readline()
It is unused and adds complexity.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 07 Oct 2015 15:33:52 -0700] rev 30686
keepalive: don't concatenate strings when reading chunked transfer
Surprisingly, this didn't appear to speed up HTTP-based stream cloning
on my machine. I suspect this has more to do with the fact we're using
small HTTP chunks and string concatenation overhead isn't so bad.
However, the reasons for this change are solid: we know string
concatenation can be a performance sink.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 26 Dec 2016 12:11:29 -0700] rev 30685
exchange: use rich class for sorting clone bundle entries
Python 3 removed the "cmp" argument from sorted(). Custom sorting in
Python 3 must be implemented with the dunder comparison methods on
types and/or with a "key" function.
This patch converts our custom "cmp" function to a custom type.
The implementation is very similar to functools.cmp_to_key(). However,
cmp_to_key() doesn't exist in Python 2, so we can't use it.
This was the only use of the "cmp" argument to sorted() in the code
base.
Kevin Bullock <kbullock@ringworld.org> [Wed, 04 Jan 2017 10:51:37 -0600] rev 30684
Added signature for changeset e69874dc1f4e
Kevin Bullock <kbullock@ringworld.org> [Wed, 04 Jan 2017 10:51:31 -0600] rev 30683
Added tag 4.0.2 for changeset e69874dc1f4e
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sat, 31 Dec 2016 17:19:09 +0900] rev 30682
i18n-ja: synchronized with 5f33116cd787
Jun Wu <quark@fb.com> [Mon, 26 Dec 2016 00:02:42 +0000] rev 30681
chg: respect XDG_RUNTIME_DIR
$XDG_RUNTIME_DIR [1] is a better place for user daemons. Let's use it and
fallback to $TMPDIR.
After this patch, chg will try socket paths in the following order:
1. $CHGSOCKNAME
2. $XDG_RUNTIME_DIR/chg/server
3. ${TMPDIR:-tmp}/chg$UID/server
[1]: https://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
Jun Wu <quark@fb.com> [Sun, 25 Dec 2016 23:49:54 +0000] rev 30680
chg: make "get default sockdir" a separate method
The logic to get a default socket directory will become longer in the next
patch. So let's move it out.
Jun Wu <quark@fb.com> [Sun, 25 Dec 2016 23:32:11 +0000] rev 30679
chg: handle connect failure before errno gets overridden
This patch moves the error handling logic up so that errno after connect
won't be overridden.
Pulkit Goyal <7895pulkit@gmail.com> [Sun, 25 Dec 2016 03:06:55 +0530] rev 30678
py3: have a bytes version of shlex.split()
shlex.split() only accepts unicodes on Python 3. After this patch we will be
using pycompat.shlexsplit(). This patch also replaces existing occurences of
shlex.split with pycompat.shlexsplit.
Jun Wu <quark@fb.com> [Fri, 23 Dec 2016 16:26:40 +0000] rev 30677
chg: support long socket path
This patch replaces UNIX_PATH_MAX (108) with PATH_MAX (4096) so we can have
long unix path.
Jun Wu <quark@fb.com> [Fri, 23 Dec 2016 16:16:44 +0000] rev 30676
chg: remove sockdirfd
See the previous patch for the reason.
Jun Wu <quark@fb.com> [Fri, 23 Dec 2016 16:37:00 +0000] rev 30675
chg: let hgc_open support long path
"sizeof(sun_path)" is too small. Use the chdir trick to support long socket
path, like "mercurial.util.bindunixsocket".
It's useful for cases where TMPDIR is long. Modern OS X rewrites TMPDIR to a
long value. And we probably want to use XDG_RUNTIME_DIR [2] for Linux.
The approach is a bit different from the previous plan, where we will have
hgc_openat and pass cmdserveropts.sockdirfd to it. That's because the
current change is easier: chg has to pass a full path to "hg" as the
"--address" parameter. There is no "--address-basename" or "--address-dirfd"
flags. The next patch will remove "sockdirfd".
Note: It'd be nice if we can use a native "connectat" implementation.
However, that's not available everywhere. Some platform (namely FreeBSD)
does support it, but the implementation has bugs so it cannot be used [2].
[1]: https://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
[2]: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-April/082892.html
Augie Fackler <raf@durin42.com> [Sat, 24 Dec 2016 15:38:27 -0500] rev 30674
rebase: un-wrap function signature since it fits in 80 columns
Pulkit Goyal <7895pulkit@gmail.com> [Sun, 25 Dec 2016 02:42:46 +0530] rev 30673
py3: exclude pywatchman from test-check-py3-compat.t
Exclude pywatchman from py3 test. They have already worked on Python 3
compatibility https://github.com/facebook/watchman/pull/247
Pulkit Goyal <7895pulkit@gmail.com> [Sun, 25 Dec 2016 02:34:19 +0530] rev 30672
py3: update test-check-py3-compat.t
This part of test runs only on py3. This change was introduced by 16f4b341288d.
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 22 Dec 2016 19:35:30 +0530] rev 30671
shelve: choose a legal shelve name when no name is passed (issue5112)
Currently if our branch name contains '\' or starts with '.', shelve chooses
an illegal shelve name. This behaviour is not good as it itself is choosing
something which it won't accept further. We can raise errors if user passes
a name which is illegal.
After this patch, if '\' is contained in branch name or bookmark name, it will
be replaced by '_' while choosing a shelve name and if they starts with '.',
the first '.' is replaced by '_'.
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 22 Dec 2016 23:27:32 +0530] rev 30670
shelve: add tests to ensure illegal shelve names are avoided
We avoid '.' as the first letter of shelve name so that we don't create hidden
file. We also avoid slashes in name so that we don't form a new directory
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 20 Dec 2016 00:20:07 +0530] rev 30669
py3: replace sys.executable with pycompat.sysexecutable
sys.executable returns unicodes on Python 3. This patch replaces occurences of
sys.executable with pycompat.sysexecutable.
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 20 Dec 2016 00:02:24 +0530] rev 30668
py3: have bytes version of sys.executable
sys.executable on Python 3 returns unicodes and we want bytes. So this patch
adds a new pycompat.sysexecutable which returns bytes by encoding using
os.fsencode() since it is path variable.
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 22 Dec 2016 01:54:17 +0530] rev 30667
py3: use pycompat.getcwd instead of os.getcwd