py3: make py3 compat.iterbytestr simpler and faster With Python 3.4.3, timit says 11.9 usec-> 6.44 usec. With Python 3.6.0, timeit says 14.1 usec -> 9.55 usec.

py3: optimize py3 compat.bytechr using Struct.pack With Python 3.4.3, timeit says 0.437 usec -> 0.0685 usec. With Python 3.6, timeit says 0.157 usec -> 0.0907 usec. So it's faster on both versions, but the speedup varies a lot. Thanks to Gregory Szorc for the suggestion.

tests: properly drop back to root dir in test-status.t

dirstate: use list comprehension to get a list of keys We have used dict.keys() which returns a dict_keys() object instead of list on Python 3. So this patch replaces that with list comprehension which works both on Python 2 and 3.

match: slice over bytes to get the byteschr instead of ascii value

match: make regular expression bytes to prevent TypeError

scmutil: make function name bytes in class filecache func.__name__ returns unicodes and this leads to keyerror when we try to do filecache[''] by passing bytes.

localrepo: deprecate 'wfile' The method had very few users and the modern form is shorter. So let us deprecates another method of the localrepo class.

eol: use 'wvfs' instead of 'wfile' Method is about to be deprecated and the modern form is shorter.

localrepo: use 'wvfs' instead of 'wfile' Method is about to be deprecated and the modern form is shorter.

tagmerge: use 'wvfs' instead of 'wfile' Method is about to be deprecated and the modern form is shorter.

gpg: use 'wvfs' instead of 'wfile' Method is about to be deprecated and the modern form is shorter.

test: add a basic 'test-check-pylint.t' We add a minimal check using pylint for one case we knows we care about: "mutable default" argument. We'll likely extend this over time to cover other useful checks but this is a good starting point.

localrepo: don't use mutable default argument value Caught by pylint.

httpclient: don't use mutable default argument value Caught by pylint.

largefiles: don't use mutable default argument value Caught by pylint.

hgk: don't use mutable default argument value Caught by pylint.

convert: don't use mutable default argument value Caught by pylint.

mq: don't use mutable default argument value Caught by pylint.

pager: if old pager extensions is enabled, respect pager.attend This patch makes us respect pager.attend again if the extension is enabled. It also brings back the default attend list, so e.g. summary is not paged by default, just like it used to be before pager was moved into core.

tests: duplicate test for pager for old extension and for in-core pager When the old pager extension is enabled, I think we should try to be as BC as reasonable. To help with that, this patch brings back test-pager.t as of 65a3b4d67a65 (pager: add a test of --pager=no functionality, 2017-02-06), but under the name test-pager-legacy.t However, since the behavior has changed in a few cases (notably by no longer respecting pager.attend), the file is modified to work with the current version. We will recover some lost BC in coming patches. Also, to make sure the in-core pager does not depend on the pager extension being enabled, this patch disables the extension in test-pager.t. It turns out that pager.attend-$cmd was only supported when the pager extension was enabled, so the tests are updated to reflect that. We will need to decide what to do with these.

util: make strdate's defaults default value a dict It was specified to be an empty list in c6adf2be6069 in 2007. It was correct at the time. But when the function was refactored in 91bc001a592f (2010), it started expecting a dict. I guess this code path is untested? Thanks to Yuya for spotting this.

py3: open file in rb mode

debuglabelcomplete: fix to call debugnamecomplete in new location debugnamecomplete was moved in a9aa67ba from commands to debugcommands, but debuglabelcomplete was not modified to call it in its new location.

tests: prove that `hg init` works with Python 3 The previous patch made `hg init` work!

pycompat: custom implementation of urllib.parse.quote() urllib.parse.quote() accepts either str or bytes and returns str. There exists a urllib.parse.quote_from_bytes() which only accepts bytes. We should probably use that to retain strong typing and avoid surprises. In addition, since nearly all strings in Mercurial are bytes, we probably don't want quote() returning unicode. So, this patch implements a custom quote() that only accepts bytes and returns bytes. The quoted URL should only contain URL safe characters which is a strict subset of ASCII. So `.encode('ascii', 'strict')` should be safe.

pycompat: alias urllib symbols directly urllib.request imports a bunch of symbols from other urllib modules. We should map to the original symbols not the re-exported ones because this is more correct. Also, it will prevent an import of urllib.request if only one of the lower-level symbols/modules is needed.

tests: clean up bad extension The presence of the "babar" extension breaks subsequent tests. So delete the file and create an empty one to return the config to sanity.

perf: perform a garbage collection before each iteration Currently, no explicit garbage collection is performed when running the microbenchmarks in `hg perf`. I think this is wrong because garbage collection can have a significant impact on execution times. And, if gc is triggered via the default heuristics, it will fire effectively randomly during subsequent benchmark iterations due to variable amount of garbage left over from previous runs. Running a gc before invoking the measured function will help ensure state is more consistent across all iterations.

formatter: support json formatting of long type By luck, we appear to not pass any long instances into the JSON formatter. I suspect this will change with all the Python 3 porting work. Plus I have another series that will convert some ints to longs that triggers this.

rebase: don't use mutable default argument value

mq: don't use mutable default argument value

util: don't use mutable default argument value I don't think this is any tight loops and we'd need to worry about PyObject creation overhead. Also, I'm pretty sure strptime() will be much slower than PyObject creation (date parsing is surprisingly slow).

match: don't use mutable default argument value There shouldn't be a big perf hit creating a new object because this function is complicated and does things that dwarf the cost of creating a new PyObject.

hgweb: don't use mutable default argument value

hgweb: don't use mutable default argument value

filemerge: don't use mutable default argument value

context: don't use mutable default argument value Mutable default argument values are a Python gotcha and can represent subtle, hard-to-find bugs. Lets rid our code base of them.

heads: enable pager

branches: enable pager

py3: fix slicing of bytes in revset.formatspec()

py3: make set of revset operators and quotes in bytes

py3: convert set of revset initial symbols back to bytes Otherwise tokenize() would fail due to comparison between unicode and bytes.

pycompat: add helper to iterate each char in bytes

branchmap: fix python 2.6 by using util.buffer() instead of passing bytearray

rebase: allow rebasing children of wd to wd if a new branch has been set (BC) The named branch of the leaf changeset can be changed by updating to it, setting the branch, and amending. But previously, there was no good way to *just* change the branch of several linear changes. If rebasing changes with another parent to '.', it would pick up a pending branch change up. But when rebasing changes that have the same parent, it would fail with 'nothing to rebase', even when the branch name was set differently. To fix this, allow rebasing to same parent when a branch has been set.

merge: check current wc branch for 'nothing to merge', not its p1 The working directory will usually be clean or very clean, and wc will usually have the same branch as its parent. This change will thus usually not make any difference and is done as a separate change to show that. It will be used in a later change.

lock: do not encode result of gethostname on Python 2 If a hostname contained non-ascii character, str.encode() would first try to decode it to a unicode and raise UnicodeDecodeError.

py3: prove `hg files --rev` works

tests: make a variable for hg binary location in test-check-py3-commands The number of which calls in here is starting to get silly.

lock: encode result of gethostname into a bytestring

config: avoid using a mutable default Nothing *currently* mutates this list, but the moment something does it'll be shared between all config instances. Avoid this eventuality.

localrepo: deprecate 'repo.join' in favor of 'repo.vfs.join' localrepo have an insane amount of method. Accessing the feature through the vfs is not really harder and allow us to schedule that method for removal.

pycompat: move imports of cStringIO/io to where they are used There's no point to import cStringIO as io since we have to select StringIO or BytesIO conditionally.

rbc: empty (and invalid) rbc-names file should give an empty name list An empty file (if it somehow should exist) used to give a list with an empty name. That didn't do any harm, but it was "wrong". Fix that.

rbc: use struct unpack_from and pack_into instead of unpack and pack These functions were introduced in Python 2.5 and are faster and simpler than the old ones ... mainly because we can avoid intermediate buffers: $ python -m timeit -s "_rbcrecfmt='>4sI'" -s 's = "x"*10000' -s 'from struct import unpack' 'unpack(_rbcrecfmt, buffer(s, 16, 8))' 1000000 loops, best of 3: 0.543 usec per loop $ python -m timeit -s "_rbcrecfmt='>4sI'" -s 's = "x"*10000' -s 'from struct import unpack_from' 'unpack_from(_rbcrecfmt, s, 16)' 1000000 loops, best of 3: 0.323 usec per loop $ python -m timeit -s "from array import array" -s "_rbcrecfmt='>4sI'" -s "s = array('c')" -s 's.fromstring("x"*10000)' -s 'from struct import pack' -s "rec = array('c')" 'rec.fromstring(pack(_rbcrecfmt, "asdf", 7))' 1000000 loops, best of 3: 0.364 usec per loop $ python -m timeit -s "from array import array" -s "_rbcrecfmt='>4sI'" -s "s = array('c')" -s 's.fromstring("x"*10000)' -s 'from struct import pack_into' -s "rec = array('c')" -s 'rec.fromstring("x"*100)' 'pack_into(_rbcrecfmt, rec, 0, "asdf", 7)' 1000000 loops, best of 3: 0.229 usec per loop

revlog: use bytes() instead of str() to get data from memoryview Fixes `files -v` on Python 3.

util: teach url object about __bytes__ __str__ tries to do something reasonable, but someone else more familiar with encoding bugs should check my work.

manifest: ensure paths are bytes (not str) in pure parser

manifest: now that node.bin is available, use it directly Previously we were getting it through revlog, which is a little unusual.

manifest: use node.bin instead of .decode('hex') The latter doesn't work in Python 3.

manifest: add __next__ methods for Python 3 Python 3 renamed .next() in the iterator protocol to __next__().

files: use native string type to load rev opt from dict

store: fix many single-byte ops to use slicing in _auxencode

py3: add "b" prefix to string literals related to module policy String literals without explicit prefix in __init__.py and policy.py are treated as unicode object on Python3, because these modules are loaded before setup of our specific code transformation (the later module is imported at the beginning of __init__.py). BTW, "modulepolicy" in __init__.py is initialized by "policy.policy". This causes issues below; - checking "policy" value in other modules causes unintentional result For example, "b'py' not in (u'c', u'py')" returns True unintentionally on Python3. - writing "policy" out fails at conversion from unicode to bytes 62939e0148f1 fixed this issue for default code path, but "policy" can be overridden by HGMODULEPOLICY environment variable (it should be rare case for developer using Python3, though). This patch does: - add "b" prefix to all string literals, which are related to module policy, in modules above. - check existence of HGMODULEPOLICY, and overwrite "policy" only if it exists For simplicity, this patch omits checking "supports_bytes_environ", switching os.environ/os.environb, and so on (Yuya agreed this in personal talking)

py3: drop unused aliases to array.array which are replaced with bytearray

pycompat: default to BytesIO instead of StringIO

repoview: specify setattr values as native strings

revlog: use bytes() to ensure text from _chunks is a reasonable type

revlog: extract first byte of revlog with a slice so it's portable

revsetlang: slice out single bytes instead of indexing For portability with Python 3.

lock: use %d to format integer into a bytestring

parser: use %d instead of %s for interpolating error position Error position is an int, so we should use %d instead of %s. Fixes failures on Python 3.

manifest: unbreak pure-python manifest parsing on Python 3

context: use portable construction to verify int parsing

ui: portably bytestring-ify url object

scmutil: fix key generation to portably bytestringify integer

branchmap: stringify int in a portable way We actually need a bytes in Python 3, and thanks to our nasty source loader this will portably do the right thing.

branchmap: don't use buffer() on Python 3 This is certainly slower than the Python 2 code, but it works, and we can revisit it later if it's a problem.

py3: use bytearray() instead of array('c', ...) constructions Portable from 2.6-3.6.

summary: don't explicitly str() something we're about to %s str() is wrong on Python 3 here, and %s implicitly calls str() anyway, so this was just extra dancing for no reason.

context: implement both __bytes__ and __str__ for Python 3 They're very similar, for obvious reasons.

context: work around `long` not existing on Python 3 I can't figure out what this branch is even trying to accomplish, and it was introduced in ac89a23ca814 which doesn't really shed any insight into why longs are treated differently from ints.

phases: explicitly evaluate list returned by map On Python 3 map() returns a generator, which bool()s to true even if it had an empty input set. Work around this by using list() on the map() result.

ui: check for --debugger in sys.argv using r-string to avoid bytes on py3 Our source loader was errantly turning this --debugger into a bytes, which was then causing me to still get a pager when I was using the debugger on py3. That made life hard.

minirst: use bytes.strip instead of str.strip bytes.strip exists in Python 2.6 and Python 2.7 also.

smcposix: pass unicode as first argument to array.array This is an instance where we can safely convert the first argument, rest are the cases except one where we are using 'c' which is not there in Python 3. So that needs to be handled differently. This will help in making `hg help` run on Python 3.

util: pass encoding.[encoding|encodingmode] as unicodes We need to pass str to encode() and decode().

win32text: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

transplant: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

shelve: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

share: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

mq: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

largefiles: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

keyword: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

journal: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

histedit: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

eol: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

convert: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

test-bundle2-remote-changegroup: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

subrepo: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

repair: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

merge: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

hg-mod: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

commands: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

cmdutil: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

localrepo: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

minirst: make encoding.encoding unicodes to pass into encode() and decode()

minirst: make regular expressions bytes

setup: convert setupversion to unicode Something deep in the bowels of distutils expects "version" passed to setup() to be a str/unicode. So, convert the type. This still works on Python 2 because the string is ascii and an implicit coercion back to str/bytes should work without issue. If it does cause problems, we can always make the unicode conversion dependent on running Python 3. This change makes `python3.5 setup.py install` work.

util: add debugstacktrace depth limit Useful when you don't care about the start of the stack, but only want to see the last entries.

util: strip trailing newline from debugstacktrace message This makes the function more convenient to use as drop-in replacement for ui.write & co.

tests: rework util.debugstacktrace tests Prepare for adding another test.

vfs: use repo.vfs.rename

vfs: use repo.vfs.unlinkpath

merge: use repo.wvfs.unlinkpath

vfs: use repo.wvfs.unlinkpath

policy: try and always have a bytes for module policy debuginstall now runs cleanly in Python 3.

init: zstd is already python3-ready, so don't run it through our importer

config: guard against setconfig specifying unicode values on py3 This was leading to some difficult to trace problems because the values were set in one place, but then blew up much later in the program. Exploding violently with an assertion seems reasonable here.