tests: add test coverage of merging x flag without ancestor It is more noisy than necessary - we will fix that later.

dirs: document Py_SIZE weirdness Assigning to what looks like a function is clown shoes. Document that it is a macro referring to a struct member.

record: return code from underlying commit

commit: return 1 for interactive commit with no changes (issue5397) For consistency with non interactive commit

demandimport: disable lazy import of __builtin__ Demandimport uses the "try to import __builtin__, else use builtins" trick to handle Python 3. External libraries and extensions might do something similar. On Fedora 25 subversion-python-1.9.4-4.fc25.x86_64 will do just that (except the opposite) ... and it failed all subversion convert tests because demandimport was hiding that it didn't have builtins but should use __builtin__. The builtin module has already been imported when demandimport is loaded so there is no point in trying to import it on demand. Just always ignore both variants in demandimport.

changelog: disable delta chains This patch disables delta chains on changelogs. After this patch, new entries on changelogs - including existing changelogs - will be stored as the fulltext of that data (likely compressed). No delta computation will be performed. An overview of delta chains and data justifying this change follows. Revlogs try to store entries as a delta against a previous entry (either a parent revision in the case of generaldelta or the previous physical revision when not using generaldelta). Most of the time this is the correct thing to do: it frequently results in less CPU usage and smaller storage. Delta chains are most effective when the base revision being deltad against is similar to the current data. This tends to occur naturally for manifests and file data, since only small parts of each tend to change with each revision. Changelogs, however, are a different story. Changelog entries represent changesets/commits. And unless commits in a repository are homogonous (same author, changing same files, similar commit messages, etc), a delta from one entry to the next tends to be relatively large compared to the size of the entry. This means that delta chains tend to be short. How short? Here is the full vs delta revision breakdown on some real world repos: Repo % Full % Delta Max Length hg 45.8 54.2 6 mozilla-central 42.4 57.6 8 mozilla-unified 42.5 57.5 17 pypy 46.1 53.9 6 python-zstandard 46.1 53.9 3 (I threw in python-zstandard as an example of a repo that is homogonous. It contains a small Python project with changes all from the same author.) Contrast this with the manifest revlog for these repos, where 99+% of revisions are deltas and delta chains run into the thousands. So delta chains aren't as useful on changelogs. But even a short delta chain may provide benefits. Let's measure that. Delta chains may require less CPU to read revisions if the CPU time spent reading smaller deltas is less than the CPU time used to decompress larger individual entries. We can measure this via `hg perfrevlog -c -d 1` to iterate a revlog to resolve each revision's fulltext. Here are the results of that command on a repo using delta chains in its changelog and on a repo without delta chains: hg (forward) ! wall 0.407008 comb 0.410000 user 0.410000 sys 0.000000 (best of 25) ! wall 0.390061 comb 0.390000 user 0.390000 sys 0.000000 (best of 26) hg (reverse) ! wall 0.515221 comb 0.520000 user 0.520000 sys 0.000000 (best of 19) ! wall 0.400018 comb 0.400000 user 0.390000 sys 0.010000 (best of 25) mozilla-central (forward) ! wall 4.508296 comb 4.490000 user 4.490000 sys 0.000000 (best of 3) ! wall 4.370222 comb 4.370000 user 4.350000 sys 0.020000 (best of 3) mozilla-central (reverse) ! wall 5.758995 comb 5.760000 user 5.720000 sys 0.040000 (best of 3) ! wall 4.346503 comb 4.340000 user 4.320000 sys 0.020000 (best of 3) mozilla-unified (forward) ! wall 4.957088 comb 4.950000 user 4.940000 sys 0.010000 (best of 3) ! wall 4.660528 comb 4.650000 user 4.630000 sys 0.020000 (best of 3) mozilla-unified (reverse) ! wall 6.119827 comb 6.110000 user 6.090000 sys 0.020000 (best of 3) ! wall 4.675136 comb 4.670000 user 4.670000 sys 0.000000 (best of 3) pypy (forward) ! wall 1.231122 comb 1.240000 user 1.230000 sys 0.010000 (best of 8) ! wall 1.164896 comb 1.160000 user 1.160000 sys 0.000000 (best of 9) pypy (reverse) ! wall 1.467049 comb 1.460000 user 1.460000 sys 0.000000 (best of 7) ! wall 1.160200 comb 1.170000 user 1.160000 sys 0.010000 (best of 9) The data clearly shows that it takes less wall and CPU time to resolve revisions when there are no delta chains in the changelogs, regardless of the direction of traversal. Furthermore, not using a delta chain means that fulltext resolution in reverse is as fast as iterating forward. So not using delta chains on the changelog is a clear CPU win for reading operations. An example of a user-visible operation showing this speed-up is revset evaluation. Here are results for `hg perfrevset 'author(gps) or author(mpm)'`: hg ! wall 1.655506 comb 1.660000 user 1.650000 sys 0.010000 (best of 6) ! wall 1.612723 comb 1.610000 user 1.600000 sys 0.010000 (best of 7) mozilla-central ! wall 17.629826 comb 17.640000 user 17.600000 sys 0.040000 (best of 3) ! wall 17.311033 comb 17.300000 user 17.260000 sys 0.040000 (best of 3) What about 00changelog.i size? Repo Delta Chains No Delta Chains hg 7,033,250 6,976,771 mozilla-central 82,978,748 81,574,623 mozilla-unified 88,112,349 86,702,162 pypy 20,740,699 20,659,741 The data shows that removing delta chains from the changelog makes the changelog smaller. Delta chains are also used during changegroup generation. This operation essentially converts a series of revisions to one large delta chain. And changegroup generation is smart: if the delta in the revlog matches what the changegroup is emitting, it will reuse the delta instead of recalculating it. We can measure the impact removing changelog delta chains has on changegroup generation via `hg perfchangegroupchangelog`: hg ! wall 1.589245 comb 1.590000 user 1.590000 sys 0.000000 (best of 7) ! wall 1.788060 comb 1.790000 user 1.790000 sys 0.000000 (best of 6) mozilla-central ! wall 17.382585 comb 17.380000 user 17.340000 sys 0.040000 (best of 3) ! wall 20.161357 comb 20.160000 user 20.120000 sys 0.040000 (best of 3) mozilla-unified ! wall 18.722839 comb 18.720000 user 18.680000 sys 0.040000 (best of 3) ! wall 21.168075 comb 21.170000 user 21.130000 sys 0.040000 (best of 3) pypy ! wall 4.828317 comb 4.830000 user 4.820000 sys 0.010000 (best of 3) ! wall 5.415455 comb 5.420000 user 5.410000 sys 0.010000 (best of 3) The data shows eliminating delta chains makes the changelog part of changegroup generation slower. This is expected since we now have to compute deltas for revisions where we could recycle the delta before. It is worth putting this regression into context of overall changegroup times. Here is the rough total CPU time spent in changegroup generation for various repos while using delta chains on the changelog: Repo CPU Time (s) CPU Time w/ compression hg 4.50 7.05 mozilla-central 111.1 222.0 pypy 28.68 75.5 Before compression, removing delta chains from the changegroup adds ~4.4% overhead to hg changegroup generation, 1.3% to mozilla-central, and 2.0% to pypy. When you factor in zlib compression, these percentages are roughly divided by 2. While the increased CPU usage for changegroup generation is unfortunate, I think it is acceptable because the percentage is small, server operators (those likely impacted most by this) have other mechanisms to mitigate CPU consumption (namely reducing zlib compression level and pre-generated clone bundles), and because there is room to optimize this in the future. For example, we could use the nullid as the base revision, effectively encoding the full revision for each entry in the changegroup. When doing this, `hg perfchangegroupchangelog` nearly halves: mozilla-unified ! wall 21.168075 comb 21.170000 user 21.130000 sys 0.040000 (best of 3) ! wall 11.196461 comb 11.200000 user 11.190000 sys 0.010000 (best of 3) This looks very promising as a future optimization opportunity. It's worth that the changes in test-acl.t to the changegroup part size. This is because revision 6 in the changegroup had a delta chain of length 2 before and after this patch the base revision is nullrev. When the base revision is nullrev, cg2packer.deltaparent() hardcodes the *previous* revision from the changegroup as the delta parent. This caused the delta in the changegroup to switch base revisions, the delta to change, and the size to change accordingly. While the size increased in this case, I think sizes will remain the same on average, as the delta base for changelog revisions doesn't matter too much (as this patch shows). So, I don't consider this a regression.

revlog: add instance variable controlling delta chain use This is to support disabling delta chains on the changelog in a subsequent patch.

changegroup: document deltaparent's choice of previous revision As part of debugging low-level changegroup generation, I came across what I initially thought was a weird behavior: changegroup v2 is choosing the previous revision in the changegroup as a delta base instead of p1. I was tempted to rewrite this to use p1, as p1 will delta better than prev in the common case. However, I realized that taking p1 as the base would potentially require resolving a revision fulltext and thus require more CPU for e.g. server-side processing of getbundle requests. This patch tweaks the code comment to note the choice of behavior. It also notes there is room for a flag or config option to tweak this behavior later: using p1 as the delta base would likely make changegroups smaller at the expense of more CPU, which could be beneficial for things like clone bundles.

help: backout f3c4edfd35e1 (mark boolean flags with [no-] in help) for now The ability to negate any boolean flags itself is great, but I think we are not ready to expose the help side of it yet. First, while there exist a handful of such flags whose default value can be changed (eg: git diff, patchwork confirmation), there is only a few of them. The users who benefit the most from this change are alias users and large installation that can deploy extension to change behavior (eg: facebook tweakdefault). So the majority of user who will be affected by a large change to command help that is not yet relevant to them. (I expect this to become relevant when ui.progressive start to exists). Below is an example of the impact of the new help on 'hg help diff': -r --rev REV [+] revision -c --change REV change made by revision -a --[no-]text treat all files as text -g --[no-]git use git extended diff format --[no-]nodates omit dates from diff headers --[no-]noprefix omit a/ and b/ prefixes from filenames -p --[no-]show-function show which function each change is in --[no-]reverse produce a diff that undoes the changes -w --[no-]ignore-all-space ignore white space when comparing lines -b --[no-]ignore-space-change ignore changes in the amount of white space -B --[no-]ignore-blank-lines ignore changes whose lines are all blank -U --unified NUM number of lines of context to show --[no-]stat output diffstat-style summary of changes --root DIR produce diffs relative to subdirectory -I --include PATTERN [+] include names matching the given patterns -X --exclude PATTERN [+] exclude names matching the given patterns -S --[no-]subrepos recurse into subrepositories Another issue with the current state of help, the default value for the flag is not conveyed to the user. For example in the 'backout' help, there is no real distinction between "--[no-]backup" (default to True) and "--[no-]keep" (default) to False: --[no-]backup no backups --[no-]keep do not modify working directory during strip In addition, I've discussed with Augie Fackler and the last batch of the work on this have burned him out quite some. Therefore he is not intending to perform any more work on this topic. Quoting him, he would rather see the help part backed out than spending more time on it. I do not think we are ready to expose this to users in 4.0 (freeze in a week), especially because we cannot expect quick improvement on these aspect as this topic no longer have an owner. We should be able to reintroduce that change in the future when someone get back on it and the main issues are solves: * Introduction of ui.progressive makes it relevant for a majority of user, * Current default value are efficiently conveyed to the user. (In addition, the excerpt from diff help show that we still have some issue with some negative option like '--nodates' so further improvement are probably welcome there.)

copy: distinguish "file exists" cases and add a hint (BC) Users that want to add a copy record to an existing commit with 'hg commit --amend' should be guided towards this workflow, rather than reaching for some sort of uncommit-recommit flow. As part of this, distinguish in the top-line error message whether the file merely already exists (untracked) on disk or the file already exists in history. The full list of copy and rename cases and how they interact with flags are listed below: target exists --after --force | action n n * | copy n y * | (1) untracked n n | (4) NEWHINT untracked n y | (3) untracked y * | (2) y n n | (4) NEWHINT y n y | (3) y y n | (2) y y y | (3) deleted n n | copy deleted n y | (3) deleted y n | (1) deleted y y | (1) * = don't care (1) <src>: not recording move - <target> does not exist (2) preserve target contents (3) replace target contents (4) <target>: not overwriting - file {exists,already committed} Credit to Kevin for wholly rewriting my table to cover more cases we discovered at the sprint. I think this change gets the hints correct in all cases, but I'd appreciate close inspection of the test cases to make sure I haven't gotten turned around in here.

perf: make perftags clear tags cache correctly Before this patch, "hg perftags" command doesn't measure performance of "repo.tags()" correctly, because it doesn't clear tags cache correctly. 9dca7653b525 replaced repo._tags with repo._tagscache, but didn't change the code path to clear tags cache in perftags() at that time. BTW, full history of "tags cache" is: - d7df759d0e97 (or 0.6) introduced repo.tagscache as the first "tags cache" - 5614a628d173 (or 1.4) replaced repo.tagscache with repo._tags - 9dca7653b525 (or 2.0) replaced repo._tags with repo._tagscache - 98c867ac1330 (or 2.5) made repo._tagscache filteredpropertycache To make perftags clear tags cache correctly, and to increase "historical portability" of perftags, this patch examines existence of attributes in repo object, and guess appropriate procedure to clear tags cache. To avoid examining existence of attributes at each repetition, this patch makes repocleartagscachefunc() return the function, which actually clears tags cache. mozilla-central repo (85 tags on 308365 revs) with each Mercurial version between before and after this patch. ==== ========= ========= ver before after ==== ========= ========= 1.9 0.476062 0.466464 ------- *1 ------- 2.0 0.346309 0.458327 2.1 0.343106 0.454489 ------- *2 ------- 2.2 0.069790 0.071263 2.3 0.067829 0.069340 2.4 0.068075 0.069573 ------- *3 ------- 2.5 0.021896 0.022406 2.6 0.021900 0.022374 2.7 0.021883 0.022379 2.8 0.021949 0.022327 2.9 0.021877 0.022330 3.0 0.021860 0.022314 3.1 0.021869 0.022669 3.2 0.021831 0.022668 3.3 0.021809 0.022691 3.4 0.021861 0.022916 3.5 0.019335 0.020749 3.6 0.019319 0.020866 3.7 0.018781 0.020251 ------- *4 ------- 3.8 0.068262 0.072558 3.9 0.069682 0.073773 ==== ========= ========= (*1) repo._tags was replaced with repo._tagscache at this point "repo._tags = None" in perftags "before" this patch doesn't clear tags cache for Mercurial 2.0 or later. This causes significant gap of "before" between 1.9 and 2.0 . (*2) I'm not sure about significant gap at this point, but release note of 2.2 described "a number of significant performance improvements for large repositories" (*3) filtered changelog was cached in repoview as repoview.changelog at this point (by 4d92e2d75cff) This avoids calculation of filtered changelog at each repetition of t(). (*4) calculation of filtered changelog was included into wall time at this point (by 332926212ef8), again See below for detail about this significant gap: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-April/083410.html

perf: replace ui.configint() by getint() for Mercurial earlier than 1.9 Before this patch, using ui.configint() prevents perf.py from measuring performance with Mercurial earlier than 1.9 (or fa2b596db182), because ui.configint() isn't available in such Mercurial, even though there are some code paths for Mercurial earlier than 1.9 in perf.py. For example, setting "_prereadsize" attribute in perfindex() and perfnodelookup() is effective only with hg earlier than 1.8 (or 61c9bc3da402). This patch replaces ui.configint() invocations by newly introduced getint(). This patch also adds check-perf-code.py an extra check entry to detect direct usage of ui.configint() in perf.py. BTW, this patch doesn't choose adding configint() method at runtime by replacing ui.__class__ like below, even though this is the recommended way to modern Mercurial extensions. def uisetup(ui): if not util.safehasattr(ui, 'configint'): class uiwrap(ui.__class__): def configint(self, section, name, ....): .... ui.__class__ = uiwrap Because changes to ui.__class__ by uisetup() of loaded extension have been propagated since 1.6.1 (or d8d0fc3988ca), the recommended way above doesn't work as expected with Mercurial earlier than it.

perf: omit copying from ui.ferr to ui.fout for Mercurial earlier than 1.9 Before this patch, referring ui.ferr prevents perf.py from measuring performance with Mercurial earlier than 1.9 (or 4e1ccd4c2b6d), because ui.ferr isn't available in such Mercurial, even though there are some code paths for Mercurial earlier than 1.9 in perf.py. For example, setting "_prereadsize" attribute in perfindex() and perfnodelookup() is effective only with hg earlier than 1.8 (or 61c9bc3da402).

perf: define formatter locally for Mercurial earlier than 2.2 Before this patch, using ui.formatter() prevents perf.py from measuring performance with Mercurial earlier than 2.2 (or ae5f92e154d3), because ui.formatter() isn't available in such Mercurial, even though there are some code paths for Mercurial earlier than 2.2 in perf.py. For example, setting "_prereadsize" attribute in perfindex() and perfnodelookup() is effective only with hg earlier than 1.8 (or 61c9bc3da402). This patch defines formatter class locally, and use it instead of the value returned by ui.formatter(), if perf.py is used with Mercurial earlier than 2.2. In this case, we don't need to think about -T/--template option for formatter, because previous patch made -T/--template disabled for perf.py with Mercurial earlier than 3.2 (or 7a7eed5176a4).

perf: add functions to get vfs-like object for Mercurial earlier than 2.3 Before this patch, using svfs prevents perf.py from measuring performance of Mercurial earlier than 2.3 (or 7034365089bf), because svfs isn't available in such Mercurial, even though there are some code paths for Mercurial earlier than 2.3 in perf.py. For example, setting "_prereadsize" attribute in perfindex() and perfnodelookup() is effective only with hg earlier than 1.8 (or 61c9bc3da402). To get appropriate vfs-like object to access files under .hg/store, this patch adds getsvfs() (and also getvfs(), for future use). To avoid examining existence of attribute at each repetition while measuring performance, getsvfs() is invoked outside the function to be called repeatedly. This patch also adds check-perf-code.py an extra check entry to detect direct usage of repo.(vfs|svfs|opener|sopener) in perf.py.

perf: avoid actual writing branch cache out correctly Mercurial 2.5 (or 9b6ae29d4801) introduced "perfbranchmap" command, and tried to avoid actual writing branch cache out by replacing write() of branchcache class in branchmap.py with no-op function (probably, for elimination of noisy and heavy file I/O factor). But its implementation isn't correct, because 9b6ae29d4801 replaced not branchmap.branchcache.write() but branchmap.write(). The latter doesn't exist, even at that change. To avoid actual writing branch cache out correctly, this patch replaces branchmap.branchcache.write() with no-op function. To detect mistake of replacement or change of API in the future quickly, this patch uses safeattrsetter() instead of direct attribute assignment. For similarity between replacements, this patch also changes replacement of branchmap.read(). In this patch, replacement of read()/write() can run safely outside "try" block, because two safeattrsetter() invocations ensure that replacement doesn't cause exception. FYI, the table below compares "base" filter wall time of perfbranchmap on recent mozilla-central repo with each Mercurial version between before and after this patch. ==== ========= ========= ver before after ==== ========= ========= 2.5 18.492334 18.232455 2.6 18.733858 18.156702 2.7 18.245598 18.349210 2.8 18.289070 18.528422 2.9 17.572742 16.989655 3.0 17.406953 17.615012 3.1 17.228419 17.689805 3.2 17.862961 17.718367 3.3 2.632110 2.707960 3.4 3.285683 3.272060 3.5 3.370141 3.352176 3.6 3.366939 3.242455 3.7 3.300778 3.367328 3.8 3.300132 3.267298 3.9 3.418996 3.370265 ==== ========= ========= IMHO, there is no serious overlooking performance regression.

perf: get subsettable from appropriate module for Mercurial earlier than 2.9 Before this patch, using branchmap.subsettable prevents perfbranchmap from measuring performance of Mercurial earlier than 2.9 (or 175c6fd8cacc), because 175c6fd8cacc moved subsettable from repoview.py to branchmap.py, even though there are some code paths for Mercurial earlier than 2.9 in perf.py. For example, setting "_prereadsize" attribute in perfindex() and perfnodelookup() is effective only with hg earlier than 1.8 (or 61c9bc3da402). To get subsettable from appropriate module, this patch examines existence of subsettable in branchmap and repoview. This patch also adds check-perf-code.py an extra check entry to detect direct usage of subsettable attribute in perf.py.

perf: introduce safeattrsetter to replace direct attribute assignment Referring not-existing attribute immediately causes failure, but assigning a value to such attribute doesn't. For example, perf.py has code paths below, which assign a value to not-existing attribute. This causes incorrect performance measurement, but these code paths are executed successfully. - "repo._tags = None" in perftags() recent Mercurial has tags cache information in repo._tagscache - "branchmap.write = lambda repo: None" in perfbranchmap() branchmap cache is written out by branchcache.write() in branchmap.py "util.safehasattr() before assignment" can avoid this issue, but might increase mistake at "copy & paste" attribute name or so. To centralize (1) examining existence of, (2) assigning a value to, and (3) restoring an old value to the attribute, this patch introduces safeattrsetter(). This is used to replace direct attribute assignment in subsequent patches. Encapsulation of restoring is needed to completely remove direct attribute assignment from perf.py, even though restoring isn't needed so often.

largefiles: use context for file closing Make the code slightly smaller and safer (and more deeply indented).

largefiles: when setting/clearing x bit on largefiles, don't change other bits It is only the X bit that it matters to copy from the standin to the largefile in the working directory. While it generally doesn't do any harm to copy the whole mode, it is also "wrong" to copy more than the X bit we care about. It can make a difference if someone should try to handle largefiles differently, such as marking them read-only. Thus, do similar to what utils.setflags does and set the X bit where there are R bits and obey umask.

eol: on update, only re-check files if filtering changed Before, update would mark all files as 'normallookup' in dirstate if .hgeol changed so all files would get the new filtering applied. That takes some time ... and is pointless if the filtering for that file didn't change. Instead, keep track of the old filtering and only check files where the filtering is changed. To keep the old filtering, change to write the applied .hgeol content to .hg/eol.cache instead of just touching it. That change is backwards/forwards compatible. In a real world test, this takes an update that is changing .hgeol and 30000 files from 12s to 4s - where the remaining eol overhead is 1-2s.

dirs: add comment about _PyBytes_Resize So readers have a canonical function to compare this code to.

checkcopies: extract the '_related' closure There is not need for it to be a closure.

checkcopies: add an inline comment about the '_related' call This helps understanding the flow of the function.

checkcopies: minor change to comment This helped me understand the refactoring so this must be helpful.

checkcopies: rename 'ca' to 'base' This variable was named after the common ancestor. It is actually the merge base that might differ from the common ancestor in the graft case. We rename the variable before a larger refactoring to clarify the situation.

bisect: extra a small initialisation outside of a loop Having initialisation done during the first iteration is cute, but can be avoided.

pycompat: only accept a bytestring filepath in Python 2

py3: use raw strings in line continuation (convert ext) Our py2 to py3 string translations marks those as bytestrings.

py3: namedtuple takes unicode (journal ext) namedtuple usage consistent with changelog.py:141

debuginstall: use %d instead of %s for formatting an int % formatting on bytes on Python 3 is pickier about which % character we specify.

py3: test to check which commands run This test helps us to keep track on the commands which runs to Python 3. The full traceback is hidden. Thanks to Augie and Martijn to wrap it up in four lines.

bisect: build a displayer only once There is multiple spot using this, building it early will help to extract more of the logic into the bisect module.

bisect: factor commonly update sequence For now, This remains a closure in the module to avoid circular import with used module.

bisect: move check_state into the bisect module Now that the function is simpler, we resume our quest to move the logic into the bisect module. In the process, we add basic documentation.

bisect: simplify conditional in 'check_state' Now that extra code about "updating" flag have been removed, we can simplify the condition flow and remove a level.

bisect: remove code about "update-flag" in check_state Now that the flag dedicated to updating the flag are handled earlier, we do not need to handle them in the 'check_state' function.

bisect: rename 'check_code' to match our naming scheme We need to do it early, otherwise 'check-commit' will complain every time we touch it.

bisect: minor movement of code handle flag updating state The code flag handling is quite complicated, we are moving code around to prepare further simplification.

test-clone: discard lock-related messages We can't predict where those will show up and they're not super-important for the contents of this particular test, so just drop them. Further reduces the flakiness of the test to zero.

test-clone: fix some instability in pooled clone race condition test Healthy output (one log file mentioning "existing pooled" and one mentioning "new pooled") will now print in a stable order, but unhealthy output will print some sort of error. This reduces the flakiness of the test from 55% to 38%. My next patch makes it completely stable.

py3: add an os.fsencode backport to ease path handling

py3: a second argument to open can't be bytes This fixes open(filename, 'r'), open(filename, 'w'), etc. calls. In Python 3, that second argument *must* be a string, you can't use bytes. The fix is the same as used with getattr() (where the second argument must also always be a string); in the tokenizer, where we detect calls, if there is something that looks like a call to open (and is not an attribute, so the previous token is not a "." dot) then make sure that that second argument is not converted to a `bytes` object instead. There is some remaining issue where the current transformer will also rewrite open(f('foo')). However this also affect function for which we perform similar rewrite ('getattr', 'setattr', 'hasattr', 'safehasattr') and will be dealt with in a follow up.

py3: make check-py3-compat.py import importlib only if necessary importlib isn't available on Python 2.6, and it isn't necessary for Py2 checks.

templater: handle division by zero in arithmetic For now, just turn it to an abort.

templater: provide arithmetic operations on integers The termwidth template keyword is of limited use without some way to ensure that margins are respected. Provide a full set of arithmetic operators (four basic operations plus the mod function, defined to match Python's // for division), so that you can create termwidth based layouts that match the user's terminal size

eol: store and reuse pattern matchers instead of creating in tight loop More "right" and more efficient.

eol: fix variable naming - call it _eolmatch instead of _eolfile It is not the file but a match object based on it.

parsers: move PyInt aliasing out of util.h The PyInt aliasing is only used by parsers.c. Since we don't want to encourage the use of PyInt parsing, move the aliasing to parsers.c.

osutil: use PyLongObject on Python 3 for listdir_slot This code looks performance sensitive. So let's retain PyIntObject on Python 2 and use PyLongObject explicitly on Python 3.

osutil: use PyLongObject in recvfds PyIntObject doesn't exist in Python 3. While PyIntObject is preferred on Python 2 because it is a fixed capacity and faster, the difference between PyIntObject and PyLongObject for scenarios where performance isn't critical or the caller isn't performing type checking shouldn't be relevant. So change recvfds to return a list of longs instead of ints on Python 2.

py3: use encoding.environ instead of os.environ This complains while running hg version on Python 3.5

store: py26 compat, don't use a dict comprehension

dirs: document performance reasons for bypassing Python C API So someone isn't tempted to change it.

dirs: port PyInt code to work on Python 3 PyIntObject no longer exists in Python 3. Instead, there is PyLongObject. Furthermore, PyInt_AS_LONG is a macro referencing a struct member. PyInt_AS_LONG doesn't exist in Python 3 and PyLong_AS_LONG is a #define for PyLong_AsLong, which is a function. So assigning to the return value of PyLong_AS_LONG doesn't work. This patch introduces a macro for obtaining the value of an integer-like type that works on Python 2 and Python 3. On Python 3, we access the struct field of the underlying PyLongObjet directly, without overflow checking. This is essentially the same as what Python 2 was doing except using a PyLong instead of a PyInt.

dirs: convert PyString to PyBytes PyStringObject was renamed to PyBytes in Python 3 along with the corresponding PyString* functions and macros. PyString* doesn't exist in Python 3. But PyBytes* is an alias to PyString in Python 2. So rewrite PyString* to PyBytes* for Python 2/3 dual compatibility.

dirs: inline string macros The old code happened to work because of how the macro was defined. This no longer works in Python 3. Furthermore, assigning to a macro just feels weird. So just inline the macro.

parsers: use PyVarObject_HEAD_INIT The macro changed slightly in Python 3, introducing curly brackets that somehow confuse Clang into issuing a ton of compiler warnings. Using PyVarObject_HEAD_INIT makes these go away. It's worth noting that the code is identical: the 2nd argument to PyVarObject_HEAD_INIT is assigned to the ob_size field and is inserted immediately after "PyObject_HEAD_INIT(type)" is generated. Compilers are weird.

pathencode: use Py_SIZE directly On Python 2, PyBytes_GET_SIZE is the same as PyString_GET_SIZE which is the same as Py_SIZE which resolves to a struct member. On Python 3, PyBytes_GET_SIZE is "(assert(PyBytes_Check(op)),Py_SIZE(op))". The compiler barfs when assigning to this version. This patch simply changes PyBytes_GET_SIZE to Py_SIZE. On Python 2, there is no effective change in behavior. On Python 3, we drop the PyBytes_Check(). However, in all cases we have explicitly created a PyBytesObject in the same function, so the PyBytes_Check() is guaranteed to be true. Despite this, code changes over time, so I've added added assert() in all callers so we can catch this in debug builds. With this patch, all mercurial.* C extensions now compile on Python 3 on my OS X machine. There are several compiler warnings and I'm sure there are incompatibilities with Python 3, including possibly segfaults. But it is a milestone.

util: remove PyString* aliases on Python 3 We no longer have any users of the legacy PyString* functions. We no longer need these redefinitions. After this change, the only reference to "PyString" in the repo is in watchman's C extension. That isn't our code and porting Mercurial extensions to Python 3 is not a high priority at the moment. watchman's C extension will be dealt with later.

parsers: convert PyString* to PyBytes* With this change, we no longer have any occurrences of "PyString" in our C extensions.

pathencode: convert PyString* to PyBytes*

osutil: convert PyString* to PyBytes* Continuing the conversion from PyString* to PyBytes*.

manifest: convert PyString* to PyBytes* Python 2.6 introduced PyBytesObject and PyBytes* as aliases for PyStringObject and PyString*. So on Python 2.6+, PyBytes* and PyString* are identical and this patch should be a no-op. On Python 3, PyStringObject is effectively renamed to PyUnicodeObject and PyBytesObject becomes the main type for byte strings. This patch begins the process of mass converting PyString* to PyBytes* so the C extensions use the correct type on Python 3.

merge: update doc of manifestmerge() per 18c2184c27dc p1 was renamed to wctx by 18c2184c27dc.

py3: remove superfluous indent from check-py3-compat.py

py3: make check-py3-compat.py load modules in standard manner Otherwise no code transformation would be applied to the modules which are imported only by imp.load_module(). This change means modules are imported from PYTHONPATH, not from the paths given by command arguments. This isn't always correct, but seems acceptable.

py3: include module filename in check-py3-compat.py output This change is intended to reduce noises in the next patch.

util: document we want Python type mapping to be temporary I think remapping Python C API types and functions is not a great approach. I'd prefer this whole #ifdef disappeared. Add a comment so we don't forget about it.

util: define PyInt_Type on Python 3 util.h attempts to wallpaper over C API differences between Python 2 and 3. This is not the correct approach where performance is critical. But it is good enough for the current state of the Python 3 port.

parsers: return NULL from PyInit_parsers on Python 3 This function must return a PyObject* or the compiler complains.

mail: take --encoding and HGENCODING into account Fall back to our encoding strategy for sending MIME text that's neither ASCII nor UTF-8.

template: provide a termwidth keyword (issue5395) We want to provide terminal-sized output. As a starting point, expose the terminal width to the templater for use in things like fill.

util: ensure forwarded attrs are set in globals() as sysstr Custom module importer strikes again.

pycompat: when setting attrs, ensure we use sysstr The custom module importer was making these bytes, so when we poked values into self.__dict__ we had bytes instead of unicode on py3 and it didn't work.

i18n: make the locale directory name the same string type as the datapath

contributing: add new file with a pointer to the wiki This also includes what I consider to be the minimum set of steps someone should be able to perform even if they can't run the testsuite. Hopefully this will help new contributors know to at least run the two checkers that find most things that (in my experience) require manual cleanup.

templater: add relpath() to convert repo path to relative path (issue5394) File paths in template are repository-absolute paths. This function can be used to convert them to filesystem paths relative to cwd. This also converts '/' to '\\' on Windows.

hgweb: fix the MRO in Python 3 object should appear at the end, otherwise it tries to pre-empt the other new-style classes in the MRO, resulting in an unresolvable MRO in Py3. We still need to include object because otherwise in 2.7 we end up with an old-style class if threading is not supported, new-style if it is.

hgweb: make fctx.annotate a separated function so it could be wrapped This patch moves "fctx.annotate" used by the "annotate" webcommand, along with the diffopts to a separated function which takes a ui and a fctx. So it could be replaced by other implementations which don't want to replace the core "fctx.annotate" directly.

zsh_completion: update some option usage flags ('+', '=' and ':') Here are the relevant symbol descriptions from [0] optspec:... The colon indicates handling for one or more arguments to the option; if it is not present, the option is assumed to take no arguments. -optname+ The first argument may appear immediately after optname in the same word, or may appear as a separate word after the option. For example, ‘-foo+:...’ specifies that the completed option and argument will look like either ‘-fooarg’ or ‘-foo arg’. -optname= The argument may appear as the next word, or in same word as the option name provided that it is separated from it by an equals sign, for example ‘-foo=arg’ or ‘-foo arg’. There are 3 types of changes in this patch: - addition of '=': means that '--repository ~/foo' can now also be spelled as '--reporitory=~/foo' - addition of '+': means '-r 0' can now also be spelled as '-r0' - removal of '+', '=' and ':': means that '-u|--untrusted' doesn't take any arguments, so '-uq' is definitely '-u -q' and not '--untrusted=q' Occasionally, ':' had to be added together with '=' and '+'. This patch is mostly just making zsh_completion file look more like zsh's own hg completion file so that we can more easily take improvements from them or send patches to them. Some context for this patch from zsh project: [2] [3]. [0]: http://zsh.sourceforge.net/Doc/Release/Completion-System.html [1]: https://sourceforge.net/p/zsh/code/ci/master/tree/Completion/Unix/Command/_hg [2]: https://sourceforge.net/p/zsh/code/ci/92584634d3d39e9ca64475ae5af8010e2ccebe24/ [3]: http://www.zsh.org/mla/workers/2015/msg02551.html

manifest: drop Py_TPFLAGS_HAVE_SEQUENCE_IN from tp_flags in Python 3 This flag disappeared in Python 3. It is only necessary in Python 2, apparently.

import: abort instead of crashing when copy source does not exist (issue5375) Previously, when a patch contained a move or copy from a source that did not exist, `hg import` would crash. This patch changes import to raise a PatchError with an explanantion of what is wrong with the patch to avoid the stack trace and bad user experience.

py3: make encodefun in store.py compatible with py3k This ensures that the filename encoding functions always map bytestrings to bytestrings regardless of python version.

py3: make the string unicode so its iterable in py3k

copies: don't record divergence for files needing no merge This is left over from when _checkcopies was factored out from mergecopies. The 2nd break has "of = None" before it, so it's a functionally equivalent change. The 1st one, however, causes a divergence to be recorded when a file has been renamed, but there is nothing to be merged to it. This is currently harmless, since the extra divergence is simply ignored later. However, the new _checkcopies introduced in the rest of this series does more than just record a divergence after completing the main loop, and it's important that the "post-processing" stage is really skipped for no-merge-needed renames.

hgweb: avoid line wrap between revision and annotate-info (issue5398) Add white-space: nowrap to td.annotate to avoid wrapping div.annotate-info into next line if there is revision number in the same cell, as it is hard to mouse over div.annotate-info if it's wrapped into next line.

py3: make format strings unicodes and not bytes Fixes issues on Python 3, wherein docstrings are unicodes. Shouldn't break anything on Python 2.

mail: handle renamed email.Header We are still using email.Header which was renamed to email.header back in Python 2.5. References: https://hg.python.org/cpython/file/2.4/Lib/email and https://hg.python.org/cpython/file/2.5/Lib/email

revset: build _syminitletters from a saner source: the string module For now, these sets will be unicode characters in Python 3, which is probably wrong, but it un-blocks importing the module so we can get further along. In the future we'll have to come up with a reasonable encoding strategy for revsets in Python 3. This patch was originally pair-programmed with Martijn.

mq: release lock after transaction in qrefresh The transaction should be closed within the lock.

perf: release lock after transaction in perffncachewrite The transaction should be closed within the lock.

pull: grab wlock during pull because pull might move bookmarks and bookmark are protected by wlock, we have to grab wlock for pull :-( This required a small upgrade of the 'lockdelay' extension used by 'test-clone.t' because the delay must apply to a single lock only.

bisect: move 'printresult' in the 'hbisect' module The logic is already extracted into a closure. We move it into the module dedicated to bisect. A minor change is applied: the creation of the 'displayer' is kept in the main command function, it remove the needs to import 'cmdutil' in 'hbisect'. This would create an import circle otherwise.

bisect: move the 'extendrange' to the 'hbisect' module We have a module ready to host any bisect logic. That logic was already isolated in a function so we just migrate it as is.

bisect: extract the 'reset' logic into its own function This is part of small clean up movement. The bisect module seem more appropriate to host the bisect logic. The cleanup itself is motivated by some higher level cleanup around vfs and locking.

bisect: access the filesystem through vfs when reseting We have nice and shiny abstractions now.

util: remove the copypasta unquote function The _urlunquote function was added back in the day to improve startup performance, but this version is a) not compatible with Python 3 and b) has quadratic performance issues that Python core solved eons ago. Moreover, the function moved from urllib to urlparse (cheaper to import) *and* that module is already imported into pycompat. As a result, removing this function improves perf now. Before: ! wall 0.066773 comb 0.000000 user 0.000000 sys 0.000000 (best of 100) After: ! wall 0.065990 comb 0.000000 user 0.000000 sys 0.000000 (best of 100)

merge: add conflict labels to merge command Now that we present the conflict labels in prompts, it's useful to have better names than "local" and "other" for every command.

check-commit: allow underscore as commit topic It's currently not possible to commit with a changeset topic like 'bash_completion'. This change fixes that.

merge: use labels in subrepo merge This is the last place that doesn't respect conflict labels in merge output. Teach subrepos to use subrepo merge output too.

registrar: make format strings unicodes and not bytes Fixes issues on Python 3, wherein docstrings are unicodes. Shouldn't break anything on Python 2.

extensions: move the "import" logic out from "load" The "load" method does too many things: on-demand import and check version. This patch moves the import logic out from "load" so it could be wrapped to change the import behavior, for example, chg will use it to pre-import extensions.

hgmanpage: stop using raw-unicode strings These don't exist in Python 3, and this ends up looking a little more explicit to Martijn and me anyway.

revset: define _symletters in terms of _syminitletters

revset: remove doubled space

util: use string.hexdigits instead of defining it ourselves This resolves some Python 3 weirdness.

util: correct check of sys.version_info sys.version is a string, and shouldn't be compared against a tuple for version comparisons. This was always true, so we were never disabling gc on 2.6. >>> (2, 7) >= '2.7' True >>> (2, 6) >= '2.7' True

py3: switch to .items() using transformer .iteritems() don't exist in Python 3 world. Used the transformer to replace .iteritems() to .items()

py3: handle multiple arguments in .encode() and .decode() There is a case and more can be present where these functions have multiple arguments. Our transformer used to handle the first argument, so added a loop to handle more arguments if present.

py3: convert to unicode to pass into encode() encoding.encoding is bytes, we need to pass it to encode() which accepts unicodes in py3, so used pycomapt.sysstr() Also this can't be done using transformer as that only transforms the string values not variables.

templater: use "needle" and "haystack" as (meta-)variables for ifcontains() It wasn't immediately clear if it's supposed to look for "search" in "thing" or "thing" in "search".

copies: mark checkcopies as internal with the _ prefix

copies: split u1/u2 to u1u/u2u and u1r/u2r These will be made different in case of grafts by another patch in this series.

copies: style fixes and add comment

copies: limit is an optimization, and doesn't provide guarantees

revset: do not rewrite ':y' to '0:y' (issue5385) That's no longer valid since the revision 0 may be hidden. Bypass validating the existence of '0' and filter it by spanset.

revset: extract function that creates range set from computed revisions So we can pass m=0 to _makerangeset() even if the revision 0 is hidden. Hidden revisions are filtered by spanset.

lazymanifest: write a more efficient, pypy friendly version of lazymanifest

hg: set default path correctly when doing a clone+share (issue5378) Before, if performing a clone+share from a repo that was itself using shared storage, the share code would copy paths.default from the underlying repo being shared, not from the source given by the user. This patch teaches hg.clonewithshare to resolve paths.default and pass it to share so it can be written to the hgrc accordingly.