tests: fix test-bundle2-format output Forgot to amend.

localrepo: recreate phasecache if changelog was modified (issue4855) Because _phaserevs and _phasesets cache revision numbers, they must be invalidated if there are new commits or stripped revisions. We could do that by calling _phasecache.invalidate(), but it wasn't simple to be integrated with the filecache mechanism. So for now, phasecache will be recreated after repo.invalidate() if 00changelog.i was modified before.

bundle2: allow compressed bundle This changeset adds support for a 'compression' parameter in bundle2 streams. When set, it controls the compression algorithm used for the payload part of the bundle2. There is currently no usage of this except in tests.

test-bundle2: dump bundle content using f --hexdump Thanks to Greg Szorc for pointing this out.

treemanifest: rework lazy-copying code (issue4840) The old lazy-copy code formed a chain of copied manifests with each copy. Under typical operation, the stack never got more than a couple of manifests deep and was fine. Under conditions like hgsubversion or convert, the stack could get hundreds of manifests deep, and eventually overflow the recursion limit for Python. I was able to consistently reproduce this by converting an hgsubversion clone of svn's history to treemanifests. This may result in fewer manifests staying in memory during operations like convert when treemanifests are in use, and should make those operations faster since there will be significantly fewer noop function calls going on. A previous attempt (never mailed) of mine to fix this problem tried to simply have all treemanifests only have a loadfunc - that caused somewhat weird problems because the gettext() callable passed into read() wasn't idempotent, so the easy solution is to have a loadfunc and a copyfunc.

manifest: rename treemanifest load functions to ease debugging I'm hunting an infinite recursion bug at the moment, and having both of these methods named just _load is muddying the waters slightly.

manifest: add id(self) to treemanifest __repr__ Also rename __str__ to __repr__ since that's what we really want for pdb.

bundlerepo: let bundle repo look in the _mancache When looking up a base revision, we were ignoring the contents that were already available in the manifest's _mancache. This patch allows us to use that data instead of reading from the revlog. This is useful in our pushrebase extension (which allows rebasing on the server side during a push) because it allows us to prefetch the bundle base manifest before aquiring the repo lock (1 second saving), which means doing less work inside the lock, which means a 20% higher commit rate.

check-seclevel: wrap entry point by function This is intended to narrow scope of local variables. The global _verbose flag will be replaced later by ui.verbose.

check-seclevel: set executable bit as it has shebang

bundle20: extract core payload generation in its own function We are about to allow compressing the core of the bundle2. So we extract the generation of this bits in its own parts to make this compression phases easier in a later changesets.

unbundle20: allow registering handlers for stream level parameters As a comment in the code have been asking for, it is now possible to register piece of code that handle parameters for the stream. I've been wondering is such function should be class methods or not. I eventually went for externally decorated methods to stick with the culture of extensibility from an extensions that apply to bundle2.

bundle2: allow to specify unsupported value on error A client may support an argument but not some of its values (eg: coming "compression" parameters). We allow this case to be carried in the exception.

bundle2: rename error exception class for unsupported feature The original name explicitly mention "Part", however it is also used outside of parts related feature. We rename from 'UnsupportedPartError' to 'BundleUnknownFeatureError' to fix this.

changegroup: use a different compression key for BZ in HG10 For "space saving", bundle1 "strip" the first two bytes of the BZ stream since they always are 'BZ'. So the current code boostrap the uncompressor with 'BZ'. This hack is impractical in more generic case so we move it in a dedicated "decompression".

monoblue: provide links to branches, tags and bookmarks by name This is adapted from cd842821db2c, that was added to paper for 3.5 release. It adds another way to refer to branches, tags and bookmarks in urls: by name. It's still possible to navigate to a specific changeset hash, but now you can get more descriptive urls straight from /summary page, for example. branchentry template (and so the whole branches table on /summary and /branches) lost the column that had a plain changeset hash, because tags and bookmarks don't have this column and also because there is already a way to address branch by its changeset hash (changeset link just next to it). Maybe we can instead bring this column with a plain changeset hash to tags and bookmarks, but this, more terse, new look feels fine.

gitweb: provide links to branches, tags and bookmarks by name This is adapted from cd842821db2c, that was added to paper for 3.5 release. It adds another way to refer to branches, tags and bookmarks in urls: by name. It's still possible to navigate to a specific changeset hash, but now you can get more descriptive urls straight from /summary page, for example. branchentry template (and so the whole branches table on /summary and /branches) lost the column that had a plain changeset hash, because tags and bookmarks don't have this column and also because there is already a way to address branch by its changeset hash (changeset link just next to it). Maybe we can instead bring this column with a plain changeset hash to tags and bookmarks, but this, more terse, new look feels fine.

cmdutil: remove HG: prefix from translation strings

i18n-zh_CN: annotate broken HG: translations

lock: recognize parent locks while acquiring This is part of a series that will allow locks to be inherited by subprocesses in limited circumstances. This patch enables the logic introduced in previous patches.

test-lock.py: fix testing for forks The earlier test worked only because the held count went up to 2, so the first release brought it down to 1. Making a copy of the lock fixes that issue.

test-lock.py: allow PID to be changed in test state This will be used in upcoming patches to create locks that appear as if they're being created by child processes.

test-lock.py: add a lock wrapper that allows faking the PID This will be used in upcoming patches to create locks that appear as if they're being created by child processes.

lock: add a wrapper to os.getpid() to make testing easier This will allow us to fake locks across processes more easily.

test-lock.py: move temp dir generation to testcase In upcoming patches we'll want to create multiple test state objects with a common test directory.

test-lock.py: copy-edit assertions about file existing Before: expected lock to exists but actually did not exists After: expected lock to exist but actually did not exist

revlog: don't flush data file after every added revision The current behavior of revlogs is to flush the data file when writing data to it. Tracing system calls revealed that changegroup processing incurred numerous write(2) calls for values much smaller than the default buffer size (Python defaults to 4096, but it can be adjusted based on detected block size at run time by CPython). The reason we flush revlogs is so readers have all data available. For example, the current code in revlog.py will re-open the revlog file (instead of seeking an existing file handle) to read the text of a revision. This happens when starting a new delta chain when adding several revisions from changegroups, for example. Yes, this is likely sub-optimal (we should probably be sharing file descriptors between readers and writers to avoid the flushing and associated overhead of re-opening files). While flushing revlogs is necessary, it appears all callers are diligent about flushing files before a read is performed (see buildtext() in _addrevision()), making the flush in _writeentry() redundant and unncessary. So, we remove it. In practice, this means we incur a write(2) a) when the buffer is full (typically 4096 bytes) b) when a new delta chain is created rather than after every added revision. This applies to every revlog, but by volume it mostly impacts filelogs. Removing the redundant flush from _writeentry() significantly reduces the number of write(2) calls during changegroup processing on my Linux machine. When applying a changegroup of the hg repo based on my local repo, the total number of write(2) calls during application of the mercurial/localrepo.py revlogs dropped from 1,320 to 217 with this patch applied. Total I/O related system calls dropped from 1,577 to 474. When unbundling a mozilla-central gzipped bundle (264,403 changesets with 1,492,215 changes to 222,507 files), total write(2) calls dropped from 1,252,881 to 827,106 and total system calls dropped from 3,601,259 to 3,178,636 - a reduction of 425,775! While the system call reduction is significant, it appears to have no impact on wall time on my Linux and Windows machines. Still, fewer syscalls is fewer syscalls. Surely this can't hurt. If nothing else, it makes examining remaining system call usage simpler and opens the door to experimenting with the performance impact of different buffer sizes.

revlog: use existing file handle when reading during _addrevision _addrevision() may need to read from revlogs as part of computing deltas. Previously, we would flush existing file handles and open a new, short-lived file handle to perform the reading. If we have an existing file handle, it seems logical to reuse it for reading instead of opening a new file handle. This patch makes that the new behavior. After this patch, revlog files are only reopened when adding revisions if the revlog is switched from inline to non-inline. On Linux when unbundling a bundle of the mozilla-central repo, this patch has the following impact on system call counts: Call Before After Delta write 827,639 673,390 -154,249 open 700,103 684,089 -16,014 read 74,489 74,489 0 fstat 493,924 461,896 -32,028 close 249,131 233,117 -16,014 stat 242,001 242,001 0 lstat 18,676 18,676 0 lseek 20,268 20,268 0 ioctl 14,652 13,173 -1,479 TOTAL 3,180,758 2,930,679 -250,079 It's worth noting that many of the open() calls fail due to missing files. That's why there are many more open() calls than close(). Despite the significant system call reduction, this change does not seem to have a significant performance impact on Linux. On Windows 10 (not a VM, on a SSD), this patch appears to reduce unbundle time for mozilla-central from ~960s to ~920s. This isn't as significant as I was hoping. But a decrease it is nonetheless. Still, Windows unbundle performance is still >2x slower than Linux. Despite the lack of significant gains, fewer system calls is fewer system calls. If nothing else, this will narrow the focus of potential areas to optimize in the future.

revlog: always open revlogs for reading and appending An upcoming patch will teach revlogs to use the existing file handle to read revision data instead of opening a new file handle just for quick reads. For this to work, files must be opened for reading as well. This patch is merely cosmetic: there are no behavior changes.

revlog: support using an existing file handle when reading revlogs Currently, the low-level revlog reading code always opens a new file handle. In some key scenarios, the revlog is already opened and an existing file handle could be used to read. This patch paves the road to that by teaching various revlog reading functions to accept an optional existing file handle to read from.

revlog: add docstring for checkinlinesize() The name is deceptive: it does more than just "check." Add a docstring to clarify what's going on.

windows: insert file positioning call between reads and writes fopen() and fdopen() have a unique-to-Windows requirement that transitions between read and write operations in files opened in modes r+, w+, and a+ perform a file positioning call (fsetpos, fseek, or rewind) in between. While the MSDN docs don't say what will happen if this is not done, observations reveal that Python raises an IOError with errno 0. Furthermore, I /think/ this behavior isn't deterministic. But I can reproduce it reliably with subsequent patches applied that open revlogs in a+ mode and perform both reads and writes. This patch introduces a proxy class for file handles opened in r+, w+, and a+ mode on Windows. The class intercepts calls and audits whether a file positioning function has been called between read and write operations. If not, a dummy, no-op seek to the current file position is performed. This appears to be sufficient to "trick" Windows into allowing transitions between read and writes without raising errors.

tests: suppress verbose output of svn transaction Subversion 1.9 shows more verbose messages than 1.8 and the tests fail because of them. These outputs are not important in our tests, so let's suppress them by -q or grep -v.

formatter: use dict.update() to set arguments passed to write functions This isn't important, but update() is better than loop in general.

formatter: verify number of arguments passed to write functions zip() takes the shortest length, which can be a source of bug.

help: unify handling of DEPRECATED/EXPERIMENTAL keywords This fixes listexts() to exclude translated "(DEPRECATED)" marker correctly. On the other hand, help_() doesn't need translated keywords, but I don't think it's worth to separate untranslated keywords just for it.

help: include parens in DEPRECATED/EXPERIMENTAL keywords In some languages that have no caps, "DEPRECATED" and "deprecated" can be translated to the same byte sequence. So it is too wild to exclude messages by _("DEPRECATED").

help: define list of keywords that should be excluded from non-verbose output This list will be reused by the other deprecated/experimental handling.

merge with stable

unbundle: cleanly abort on unknown bundle2 feature The exception wasn't caught and resulted in a traceback.

phases: return zero for no-op operations (issue4751) (BC) It is rather unhelpful to return 1 if there were no changes because the request matches the current state of phases. So we just undo that.

help: add config.troubleshooting section at the top

dispatch: don't stack trace on commands like `hg .log` This used to stack trace because it raised a util.Abort which wasn't handled in this block. We now handle it. Additionally, we error out earlier instead of plodding on and showing the "log" entry of the plain `hg help` output.

gitweb, monoblue: port highlighting linked lines from paper This is adapted from 41c4bdd1d585, fbfe5748ef3e and f3393d458bf5.

gitweb, monoblue: fix vertical align of spans in .sourcelines Empty lines in file view could produce an inexplicable margin before the next line (most noticeable in browsers on webkit/blink engine). That was making empty lines seem taller than the rest. Instead of using default vertical align, let's set it to 'top'. This issue is actually present in paper, and only recently got into gitweb (2239626369f5) and monoblue (119202d4d7a4). There's a bit more to it in paper, so that will be dealt with in a future patch. Recipe to see live: preferably using a webkit/blink browser, such as chromium, browse a file with empty lines, e.g. https://selenic.com/hg/file/3.5/README#l8 Selecting a block of text that includes empty lines will reveal white "breaks" in the selection. Highlighted line (#l8) also shows such a break below itself.

merge: fix mergestate comment

rebase: avoid losing branch commits with --keepbranch (issue4835)

lock.release: do not unlink inherited locks This is part of a series that will allow locks to be inherited by subprocesses in limited circumstances. A subprocess unlinking a lock will lead to potential corruption from other concurrent processes.

lock: add a method to reacquire the lock after subprocesses exit This is part of a series that will allow locks to be inherited by subprocesses in limited circumstances.

lock: add a method to prepare the lock for inheritance This is part of a series that will allow locks to be inherited by subprocesses in limited circumstances.

lock: introduce state to keep track of inheritance This is part of a series that will allow locks to be inherited by subprocesses in limited circumstances. In upcoming patches we will refer to this state.

error: add an exception to indicate lock inheritance API contract violations

clone: check update rev for being True In 30be3aeb5344, there was an attempt to fallback to looking up the update revision assuming it was always a rev but the documentation states: True means update to default rev, anything else is treated as a revision Therefore, we should only fallback to looking up the update rev if it is not True. This bug was found in hg-git and I couldn't think of a test that does this in pure Mercurial since the source repository is checked for the revision as well (and therefore gracefully falls back).

clone: fix over-indented continuation line

resolve: consistently describe re-merge + unresolved

mercurial: add debugextensions command (issue4676) Add debugextensions command to help users debug their extension problems. If there are no extensions command prints nothing, otherwise it prints names of extension modules. If quiet or verbose option is not specified it prints(after extensions name) last version of mercurial in which given module was tested for non internal modules or not tested with user mercurial version. If verbose is specified it prints following information for every extension: extension name, import source, testedwith and buglink information. Extensions are printed sorted by extension name.

dispatch: stop warning about EPIPE in --debug mode It seems silly for "hg --debug manifest | less" to print a scary message after the user hits "q" in less. hg should just exit silently instead, since EPIPE on stdout is a perfectly reasonable result.

rebase: don't rebase obsolete commit whose successor is already rebased This patch avoids unnecessary conflicts to resolve during rebase for the users of changeset evolution. This patch modifies rebase to skip obsolete commits if they are being rebased on their successors. It introduces a new rebase state 'revprecursor' for these revisions that are being skipped and a new message to inform the user of what is happening. This feature is gated behind the config flag experimental.rebaseskipobsolete When an obsolete commit is skipped, the output is: not rebasing 14:9ad579b4a5de "I", already in destination as 17:fc37a630c901 "K"

check-code: forbid mutable value for default argument default value are common to all call. Using mutable value is a classical source of bug in Python. We forbid it. The regexp (Courtesy of Matt Mackall) is only catching such value on the first line of a definition, but that will be good enough for now.

httpconnection: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

transplant: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

mq: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs. The underlying function already handle "None" as an option value, so we do not need to do anything else.

largefiles: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

largefiles: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

largefiles: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

largefiles: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

largefiles: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

largefiles: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

largefiles: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

largefiles: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

largefiles: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

histedit: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

templater: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

templater: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

templater: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

templater: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

templater: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

addremove: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

match: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs. We added support for None in the underlying function in the parent commit. therefore we do not need to do anything but passing None.

match: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

match: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

match: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

addremove: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

tag: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs. We just added support of None for the underlying function, so nothing else the required.

commit: remove a mutable default argument Mutable default arguments are know to the state of California to cause bugs.

lock: move acquirefn call to inside the lock We're going to need to call it again as part of reinitialization after a subprocess inherits the lock.

localrepo: call lock.lock with releasefn as a keyword arg We'll be adding an acquirefn argument soon, and this makes that clearer.

copies: move debug statement to appropriate place

bidmerge: choose shortest list of diverge and rename/delete warnings Slightly less arbitrary than choosing the first set.

copies: rename diverge2 to divergeset for clarity

copies: begin separating mergecopies sides

copies: rename ctx() to getfctx() for clarity

monoblue: port code selection without line numbers from gitweb This is adapted from 2239626369f5. It also fixes issue4790 in monoblue; tab characters now have meaningful width on the modified pages (file view, file diff, changeset).

help: copy-edit description of ui.supportcontact

ui: avoid mutable default arguments I almost introduced a bug around this code by accidentally mutating a default argument. There's no reason for these to exist. It is OK to not assign {} to environ in ui.system because util.system knows how to deal with that.

util: avoid mutable default arguments I almost introduced a bug around this code by accidentally mutating a default argument. There's no reason for these to exist.

obsstore: fast path to check if obsstore is empty

obsstore: delay loading markers from obsstore file This will allow us to use cached revisions without parsing obsstore. Because _version isn't determined at __init__, the debugobsconvert command no longer works correctly. I'll fix it by a separate patch for the evolve extension.

obsstore: initialize _all markers without using _addmarkers() The next patch will make _all variable propertycached to avoid costly parsing of obsstore. This means we can't call _addmarkers() to initialize _all. Because all cached markers depend on _all, it isn't necessary to update these caches when _all is initially loaded.

revset: avoid implicit None testing in revset Implicit None testing is a very good way to get in trouble. We explicitly test for None.

revset: speed up existence checks for ordered filtered sets Previously, calling 'if foo:' on a ordered filtered set would start iterating in whatever the current direction was and return if a value was available. If the current direction was ascending, but the set had a fastdesc available, this meant we did a lot more work than necessary. If this was applied without my previous max/min fixes, it would improve max() performance (this was my first attempt at fixing the issue). Since those previous fixes went in though, this doesn't have a visible benefit in the benchmarks, but it does seem clearly better than it was before so I think it should still go in.

revset: remove existence check from min() and max() min() and max() would first do an existence check. Unfortunately existence checks can be slow in certain situations (like if the smartset is a list, and quickly iterable in both ascending and descending directions, then doing an existence check will start from the bottom, even if you want to check the max()). The fix is to not do the check, and just handle the error if it happens. In a large repo, this speeds up: hg log -r 'max(parents(. + .^) - (. + .^) & ::master)' from 3.5s to 0.85s. That revset is contrived and just for testing. In our real case we used 'bundle()' in place of '. + .^' Interesting perf numbers for the revset benchmarks: max(draft() and ::tip) => 0.027s to 0.0005s max(author(lmoscovicz)) => 2.48s to 0.57s min doesn't show any perf changes, but changing it as well will prevent a perf regression in my next patch. Result from revset benchmark revset #0: draft() and ::tip min max 0) 0.001971 0.001991 1) 0.001965 0.000428 21% revset #1: ::tip and draft() min max 0) 0.002017 0.001912 1) 0.001896 94% 0.000421 22% revset #2: author(lmoscovicz) min max 0) 1.049033 1.358913 1) 1.042508 0.319824 23% revset #3: author(lmoscovicz) or author(mpm) min max 0) 1.042512 1.367432 1) 1.019750 0.327750 23% revset #4: author(mpm) or author(lmoscovicz) min max 0) 1.050135 0.324924 1) 1.070698 0.319913 revset #5: roots((tip~100::) - (tip~100::tip)) min max 0) 0.000671 0.001018 1) 0.000605 90% 0.000946 92% revset #6: roots((0::) - (0::tip)) min max 0) 0.149714 0.152369 1) 0.098677 65% 0.100374 65% revset #7: (20000::) - (20000) min max 0) 0.051019 0.042747 1) 0.035586 69% 0.016267 38%

update: move default destination into a revset This is another step toward having "default" destination more clear and unified. Not all the logic is there because some bookmark related computation happened elsewhere. It will be moved later. The function is private because as for the other ones, cleanup is needed before we can proceed.

merge: move default destination computation in a revset This is another step toward having "default" destination more clear and unified.

mergecmd: simplify conditional The previous if tested the same thing with 'if repo._activebookmark'. We make the if/else logic clearer before a bigger refactoring.

rebase: move destination computation in a revset This is the first step toward making the "default destination" logic more clear and unified. The revset is private because I'm happy to delay the bikeshedding until after the clean up happened.

debian: include bash completions in mercurial-common

debian: install hgk as part of mercurial-common (issue4829) Leaving the hgk binary in /usr/bin causes some lintian warnings, and downstream packages poke it in /usr/share/mercurial, so we'll just stash it in there. Rather than patch hgk.py as part of the Mercurial install, just drop a config file in /etc/mercurial/hgrc.d that points to the installed hgk.

debian: install config files as part of mercurial-common Future patches will start putting config files in /etc/mercurial/, this just installs them.

debian: configuration so that cacerts is properly loaded This will be included in the mercurial-common package in a followup patch.

debian: install hg-ssh to /usr/bin just like downstream

highlight: remove temporary hack to overwrite repo.getcwd Now hgweb does it globally.

hgweb: overwrite cwd to resolve file patterns relative to repo (issue4568) It's useless to handle file patterns as relative to the cwd of the server process. The only sensible way in hgweb is to resolve paths relative to the repository root. It seems dirstate.getcwd() isn't used to get a real file path, so this patch won't cause problem.

dirstate: state that getcwd() shouldn't be used to get real file path hgweb will force it to be '' so that file patterns can be resolved relative to the repository root. I want to clarify that is correct.

merge: move merge step to the end Resolving other conflicts before merge ones is better because the state before the merge is as consistent as possible. It will also help with future work involving automatic resolution of merge conflicts with an external merge driver. There are no ordering issues here because it is easy to verify that the same file is never in both the dg/dm and the m sets.

lock: factor out lock testing into a separate function This is going to be needed for upcoming work with lock inheritance.

lock: factor code to read lock into a separate function This is going to be needed for upcoming work with lock inheritance.

tests: add unit tests for locking code We're going to make significant changes to lock behavior soon.

gitweb: port code selection without line numbers from paper This is adapted from f2e4fdb3dd27 and e92d4b8530cb. It also fixes issue4790 in gitweb; tab characters now have meaningful width on the modified pages (file view, file diff, changeset).

obsolete: remove unused _knownrevs function The call site was removed at cd62532c62a1.