changegroup: store old heads as a set Previously, the "oldheads" variable was a list. On a repository at Mozilla with 46,492 heads, profiling revealed that list membership testing was dominating execution time of applying small changegroups. This patch converts the list of old heads to a set. This makes membership testing significantly faster. On the aforementioned repository with 46,492 heads: $ hg unbundle <file with 1 changeset> before: 18.535s wall after: 1.303s Consumers of this variable only check for truthiness (`if oldheads`), length (`len(oldheads)`), and (most importantly) item membership (`h not in oldheads` - which occurs twice). So, the change to a set should be safe and suitable for stable. The practical effect of this change is that changegroup application and related operations (like `hg push`) no longer exhibit an O(n^2) CPU explosion as the number of heads grows.

checkheads: extract obsolete post processing in its own function The checkheads function is long and complex, extract that logic in a subfunction is win in itself. As the comment in the code says, this postprocessing is currently very basic and either misbehave or fails to detect valid push in many cases. My deeper motive for this extraction is to be make it easier to provide extensive testing of this case and strategy to cover them. Final test and logic will makes it to core once done.

tests: make test-simplekeyvaluefile.py py2.6-compatible Python 2.6 unittest.TestCase does not have assertRaisesRegexp.

similar: use cheaper hash() function to test exact matches We just need a hash table {fctx.data(): fctx} which doesn't keep fctx.data() in memory. Let's simply use hash(fctx.data()) to put data out from memory, and manage collided fctx objects by list. This isn't significantly faster than using sha1, but is more correct as we know SHA-1 collision attack is getting practical. Benchmark with 50k added/removed files, on tmpfs: $ hg addremove --dry-run --time -q previous: real 12.420 secs (user 11.120+0.000 sys 1.280+0.000) this patch: real 12.350 secs (user 11.210+0.000 sys 1.140+0.000)

similar: take the first match instead of the last It seems more natural. This makes the next patch slightly cleaner.

similar: do not look up and create filectx more than once Benchmark with 50k added/removed files, on tmpfs: $ hg addremove --dry-run --time -q previous: real 16.070 secs (user 14.470+0.000 sys 1.580+0.000) this patch: real 12.420 secs (user 11.120+0.000 sys 1.280+0.000)

similar: use common names for changectx variables We generally use 'wctx' and 'pctx' for working context and its parent respectively.

similar: get rid of quadratic addedfiles.remove() Instead, build a set of files to be removed and recreate addedfiles only if necessary. Benchmark with 50k added/removed files, on tmpfs: $ hg addremove --dry-run --time -q original: real 16.550 secs (user 15.000+0.000 sys 1.540+0.000) previous: real 16.730 secs (user 15.280+0.000 sys 1.440+0.000) this patch: real 16.070 secs (user 14.470+0.000 sys 1.580+0.000)

similar: sort files not by object id but by path for stable result Perhaps the original implementation would want to sort added/removed files alphabetically, but actually it did sort fctx objects by memory location. This patch removes the use of set()s in order to preserve the order of added/removed files. addedfiles.remove() becomes quadratic, but its cost appears not dominant. Anyway, the quadratic behavior will be eliminated by the next patch. Benchmark with 50k added/removed files, on tmpfs: $ mkdir src $ for n in `seq 0 49`; do > mkdir `printf src/%02d $n` > done $ for n in `seq 0 49999`; do > f=`printf src/%02d/%05d $(($n/1000)) $n` > dd if=/dev/urandom of=$f bs=8k count=1 status=none > done $ hg ci -qAm 'add 50k files of random content' $ mv src dest $ hg addremove --dry-run --time -q original: real 16.550 secs (user 15.000+0.000 sys 1.540+0.000) this patch: real 16.730 secs (user 15.280+0.000 sys 1.440+0.000)

debugfsinfo: print fstype information Since we have osutil.getfstype, it'll be handy if "debugfsinfo" prints it.

util: enable hardlink for copyfile This patch removes the global variable "allowhardlinks" that disables hardlink in all cases, so hardlink gets enabled if the filesystem type is whitelisted. Third party extensions wanting to enable hardlink support unconditionally can replace "_hardlinkfswhitelist.__contains__".

hghave: add a check about whitelisted filesystem that supports hardlink This is needed for the test added by the next patch.

util: disable hardlink for copyfile if fstype is outside a whitelist Since osutil.getfstype is available, use it to detect filesystem types. The whitelist currently includes common local filesystems on Linux where they should have good hardlink support. We may add new filesystems for other platforms later.

revlog: use pycompat.maplist to eagerly evaluate map on Python 3 According to Pulkit, this should fix `hg status --all` on Python 3.

py3: stop exporting urlparse from pycompat and util (API) There are no consumers of this in tree. Functions formerly available on this object/module can now be accessed via {pycompat,util}.urlreq.

check-code: recommend util.urlreq when importing urlparse

tests: use urlreq in tinyproxy.py This is our last consumer of util.urlparse.

bugzilla: use util.urlreq.urlparse And stop saving a module variable because it shouldn't be necessary.

pycompat: define urlreq.urlparse and urlreq.unparse aliases Currently, we export urlparse via util.urlparse then call util.urlparse.urlparse() and util.urlparse.urlunparse() in a few places. This is the only url* module exported from pycompat, making it a one-off. So let's transition to urlreq to match everything else. Yes, we double import "urlparse" now on Python 2. This will be cleaned up in a subsequent patch. Also, the Python 3 functions trade in str/unicode not bytes. So we'll likely need to write a custom implementation that speaks bytes. But moving everyone to an abstracted API is a good first step.

pycompat: remove urlunquote alias It is duplicated by urlreq.unquote and is unused. Kill it. We retain the imports because it is re-exported via util.urlparse, which is used elsewhere. Since we no longer access attributes of urlparse at module load time, this change /should/ result in that module reverting to a lazy module.

util: use urlreq.unquote pycompat.urlreq.unquote and pycompat.urlunquote effectively alias the same thing. pycompat.urlunquote is only used once in the code base. So let's switch to urlreq.unquote. "Effectively" in the above paragraph is because pycompat.urlreq.unquote aliases urllib.unquote and pycompat.urlunquote aliases urlparse.unquote on Python 2. You might think one of urllib.unquote and urlparse.unquote is an alias to the other, but you would be incorrect. In fact, these functions are copies of each other. There is even a comment in the CPython source code saying to keep them in sync. You can't make this up.

pycompat: alias urlreq.unquote to unquote_to_bytes Previously, urlreq.unquote aliased to urllib.parse.unquote, which returned a str/unicode. We like bytes, so switch urlreq.unquote to dispatch to urllib.parse.unquote_to_bytes. This required a minor helper function to register an alias under a different name from which it points. If this turns into a common pattern, we could likely teach _registeralias to accept tuple values defining the mapping. Until then, I didn't feel like adding complexity to _registeralias.

revsetlang: portably turn int into bytestring

osutil: export a "getfstype" method This patch exports the "getfstype" method. So we can use it to enable hardlinks for known safe filesystems. The patch was tested manually via debugshell on a Linux system. "mercurial.osutil.getfstype" works as expected. It's hard to mount filesystem on user-space easily. I will add a test for real hardlink support to indirectly test this patch, after turning on real hardlinks support for certain whitelisted filesystems.

osutil: add a C function getting filesystem type Currently it only has Linux filesystems, according to my Linux manpage, built at 2016-03-15. The code uses "if" instead of "switch" because there could be some duplicated values.

setup: test some header files The next patch will use "statfs", which requires different header files on different platforms. Linux: sys/vfs.h or sys/statfs.h FreeBSD or OSX: sys/param.h and sys/mount.h Therefore test them so we can include the correct ones.

setup: detect statfs statfs is not defined by POSIX but is available in various systems to help decide filesystem type. Let's detect it and set the macro HAVE_STATFS.

setup: add a function to test header files

setup: split "hasfunction" to test arbitrary code The next patch wants to test include files.

rebase: add flag to require destination In some mercurial workflows, the default destination for rebase does not always work well and can lead to confusing behavior. With this flag enabled, every rebase command will require passing an explicit destination, eliminating this confusion.