doc: add execute bit and fix shbang line for gendoc.py

contrib: add execute bit for fixpax.py

contrib: add execute bit for check-py3-compat.py

perf: make start revision configurable for perfrevlog This will help isolate performance characteristics of delta chains.

perf: use standard arguments for perfrevlog We have a convention of using -c|-m|FILE elsewhere for reading from revlogs. Use it for `hg perfrevlog`. While I was here, I also added a docstring to document what this command does, as "perfrevlog" is ambiguous.

test-extension: do not depend on demandimport (issue5012) When demandimport is disabled, the test will fail because extroot/foo.py uses import outside PYTHONPATH. This is bad since we have a PyPy migration plan and it does not support demandimport. Fix by setting PYTHONPATH.

commands: split notes into note containers

remove: quote --force in never deletes note Split Note into a note container

import: reword no hunks partial note Split Note into a note container

merge: reword help to use See help resolve

py3compat: use absolute_import

patch: use absolute_import

mdiff: use absolute_import

scmposix: use absolute_import

scmutil: use absolute_import

scmwindows: use absolute_import

store: use absolute_import

help: use absolute_import

test-commit-interactive: updates for the no-execbit case This goes with c84a07530040 and 14b184263846.

test-fileset: conditionalize output with symlink

commands: the first word of each note should be capital or `hg`

revlog: avoid string slice when decompressing u* chunks Revlog chunks can be stored uncompressed. If the first byte of the raw data is \0, we store the data as is. Else we prefix it with 'u'. Before, we performed a string slice to strip out the 'u' prefix. With this patch, we use a buffer to avoid an extra memory copy and associated garbage collection overhead. I was unable to verify any performance impact of this patch. For both mozilla-central and the hg repos, the number of manifest revisions with 'u' prefixes is very small - under 1%. So this change likely isn't called enough to have an impact on manifest reading. However, the reasoning behind this change is solid, so it should be safe.

osutil: implement pure version of recvfds() for PyPy This is less portable than the C version, but PyPy can't load CPython extensions. So for now, this will be used on PyPy. I've tested it on Linux amd64 and Mac OS X.

osutil: implement recvmsg() of SCM_RIGHTS for chg command server It will be used to attach client's stdio files to a background chg command server. The socket module of Python 2.x doesn't provide recvmsg(). This could be implemented by using ctypes, but it would be less portable than the C version because the handling of socket ancillary data heavily depends on preprocessor. Also, some length fields are wrongly typed in the Linux kernel.

cleanup: back out performance hacks amended into previous commit

commands: consistently indent notes 3 spaces most notes have 3 spaces for indentation, these had 2...

perf: add perfrevlogrevision As part of investigating performance improvements to revlog reading, I needed a mechanism to measure every part of revlog reading so I knew where time was spent and how effective optimizations were. This patch implements a perf command for benchmarking the various stages of reading a single revlog revision. When executed against a manifest revision at the end of a 30,000+ long delta chain in mozilla-central, the command demonstrates that ~80% of time is spent in zlib decompression.

commands: use revlog._deltachain in debugdeltachain We have a nice API now. Use it. This does mean we introduce an extra index lookup for each revision. Considering this is a debug command, the overhead should be acceptable. We could add the chain size to revlog._deltachain(). However, that feels like avoidable overhead.

revlog: refactor delta chain computation into own function This code is already written in multiple locations. While this code needs to be fast and extracting it to its own function adds overhead, code paths reading delta chains typically read, decompress, and do binary patching on revlog data from the delta chain. This other work (especially zlib decompression) almost certainly accounts for a lot more time than the overhead of introducing a Python function call. So I'm not worried about the performance impact of this change.

perf: call clearcaches() in perfmanifest The old code only partially cleared the caches. Now that we have a comprehensive method for wiping all caches, let's call it. This appears to introduce a marginal regression in `hg perfmanifest` on mozilla-central. This is good because the new result is more accurate since caches aren't being used.

manifest: implement clearcaches() The manifest implements its own caches in addition to revlog's. Extend the base clearcaches() to wipe these as well.

revlog: make clearcaches() more effective clearcaches() was added several years ago in e8d37b78acfb as part of implementing a perf command. Since revlog instances have many caches and since the spirit of this mostly unused method is to facilitate performance testing, I think it's appropriate for all the revlog's caches to get cleared when it is called.

fileset: detect unintentional existing() invocation at runtime A fileset predicate can invoke 'matchctx.existing()' successfully, even if it isn't marked as "existing caller". It is aborted only in some corner cases: e.g. there were one deleted file in the working directory (see 8a0513bf030a for detail). This patch makes 'matchctx.existing()' invocation abort if not '_existingenabled', which is true only while "existing caller" running. After this changes, non-"existing caller" predicate function is aborted immediately, whenever it invokes 'matchctx.existing()'. This prevent developer from forgetting to mark a predicate as "existing caller". BTW, unintentional 'matchctx.status()' invocation can be detected easily without any additional trick like this patch, because it returns 'None' if a predicate isn't marked as "status caller", and referring field (e.g. '.modified') of it is always aborted.

fileset: use set instead of list to mark predicates for efficiency (API) This reduces cost of examining whether given predicate calls 'matchctx.status()' or 'matchctx.existing()' in 'getfileset()' at runtime. This kind of examination is used also in subsequent patch, which detects unintentional 'matchctx.existing()' invocation per each predicate evaluation.

fileset: use decorator to mark a predicate as "existing caller" This can localize changes for adding (or removing) an "existing caller" predicate function in source code.

fileset: use decorator to mark a predicate as "status caller" Before this patch, predicates calling 'matchctx.status()' are listed up by immediate list value in 'getfileset()'. This prevents 3rd party extensions from adding specific predicate calling 'matchctx.status()'. This uses decorator to mark a predicate as "status caller". This can also localize changes for adding (or removing) a "status caller" predicate function in source code.

fileset: use decorator to mark a function as fileset predicate Using decorator can localize changes for adding (or removing) a fileset predicate function in source code. It is also useful to pick predicates up for specific purpose. For example, subsequent patches marks predicates as "call status" or "use existing" via decorator. To avoid (1) redundancy between "predicate name" and (the beginning of) help document, and (2) accidental typo of help document, this patch also makes decorator put predicate declration into the beginning of help.

fileset: treat encoding and eol as the predicate calling _existing Before this patch, predicate function 'encoding' and 'eol' aren't listed up in '_existingcallers', even though they invoke 'existing()'. This causes unexpected failure of these predicate, if there is a (manually) deleted file in the working directory. 8a0513bf030a and 3ce3f2b059a1 seem to overlook putting already existing 'encoding' or newly introduced 'eol' into '_existingcallers'. This patch also changes order of fileset "eol(unix)" output in test, because "existing caller" predicates show "A(dded)" files before "C(lean)" ones.

rebase: mention conflict in documentation instead of merge

rebase: simplify documentation about heads

rebase: simplify documentation about --keep Also include a warning about bookmarks

rebase: simplify documentation about selecting commits to rebase

rebase: simplify documentation about public commits add reference to graft

verify: clean up weird error/warning lists Nested functions in Python are not able to assign to variables in the outer scope without something like the list trick because assignments refer to the inner scope. So, we formerly used a list to give an object to assign into. Now that error and warning are object members, the [0] hack is no longer needed.

diff: clarify comparison as first parent

histedit: add progress support

verify: remove unreachable code to reraise KeyboardInterrupt KeyboardInterrupt should never be caught as it doesn't inherit Exception in Python 2.5 or later. And if it was, "interrupted" would be printed twice. https://docs.python.org/2.7/library/exceptions.html#exception-hierarchy

merge with stable

verify: move exc() function onto class This is part of an effort to make verify more modular so extensions can hook into it.

verify: move err() to be a class function This is part of an effort to make it easier for extensions to hook into verify.

verify: move warn() to a class level function This is part of the effort to make verify more modular so extensions can hook into it more easily.

verify: move fncachewarned up to a class variable This is part of making verify more modular so hooks can extend it.

verify: move widely used variables into class members This will allow us to start moving some of the nested functions inside verify() out onto the class. This will allow extensions to hook into verify more easily.

verify: move verify logic into a class In order to allow extensions to hook into the verification logic more easily, we need to refactor it into multiple functions. The first step is to move it to a class so the shared state can be more easily accessed.

test-install: perform the wix checking on wdir() instead of "." This allows catching problems before they are committed.

revlog: fix bad indentation (replace tab by space)

log: speed up hg log <file|folder> This patch makes hg log <file|folder> faster by using changelog.readfiles instead of changelog.read. On our large repos for hg log <file|folder> -l5 operations that were taking: - ~8s I see a 25% improvement - ~15s, I see a 35% improvement For recently modified folder/file, the difference is negligible as we don't have to consider many revisions.

changelog: add a new method to get files modified by a changeset This patch adds a new method "readfiles" to get the files modified by a changeset. It extracts some logic from "read" to only return the files modified by a changeset as efficiently as possible. This is used in the next patch to speed up hg log <file|folder>

tests: convert directory separators to '/' for MSYS in test-check-py-compat This is the same fix as 8c1d7a0e737b.

tests: make `pwd` URL compatible on Windows in test-default-push Without this, the test fails with: $ hg -q commit -A -m 'add pushurl' abort: file:// URLs can only refer to localhost $ hg push abort: file:// URLs can only refer to localhost The variable $PWD causes check-code to complain, so avoid that.

windows: correct the import of win32 This module is relative, and was overlooked when converting to absolute_import in 6daa795ed32f.

win32: add internals help topics to Inno Setup installer

changegroups: add documentation for cg3

changegroup: add flags field to cg3 delta header This lets revlog flags be transmitted over the wire. Right now this is useful for censored nodes and for narrowhg's ellipsis nodes.

changegroup: introduce cg3, which has support for exchanging treemanifests I'm not entirely happy with using a trailing / on a "file" entry for transferring a treemanifest. We've discussed putting some flags on each file header[0], but I'm unconvinced that's actually any better: if we were going to add another feature to the cg format we'd still be doing a version bump anyway to cg4, so I'm inclined to not spend time coming up with a more sophisticated format until we actually know what the next feature we want to stuff in a changegroup will be. Test changes outside test-treemanifest.t are only due to the new CG3 bundlecap showing up in the wire protocol. Many thanks to adgar@google.com and martinvonz@google.com for helping me with various odd corners of the changegroup and treemanifest API. 0: It's not hard refactoring, nor is it a lot of work. I'm just disinclined to do speculative work when it's not clear what the customer would actually be.

merge with stable

revlog: seek to end of file before writing (issue4943) Revlogs were recently refactored to open file handles in "a+" and use a persistent file handle for reading and writing. This drastically reduced the number of file handles being opened. Unfortunately, it appears that some versions of Solaris lose the file offset when performing a write after the handle has been seeked. The simplest workaround is to seek to EOF on files opened in a+ mode before writing to them, which is what this patch does. Ideally, this code would exist in the vfs layer. However, this would require creating a proxy class for file objects in order to provide a custom implementation of write(). This would add overhead. Since revlogs are the only files we open in a+ mode, the one-off workaround in revlog.py should be sufficient. This patch appears to have little to no impact on performance on my Linux machine.

manifestmerge: have manifest do matching before diffing This means that the diff code does less work, potentially significantly less in the case of treemanifests. It also should ease implementation with narrowed clone cases (such as narrowhg) when we don't always have the entire set of treemanifest revlogs locally. As far as I can tell, this codepath is currently only used by record, so it'll probably die in the near future, and then narrowhg won't have to worry about composing with some unknown matching system.

branch: reword help text We're not necessarily talking about *the* active branch, just any old branch.

archive: adjust help text

annotate: add missing period to help

addremove: make help match add

add: mention .hgignore in help

bundle: warn for --base with --all

bundle: fix error for --all with destination Before it complained about --base

bundle: fix grammar in help text

bundle: clarify help text The file might not be compressed; the interactions between -a, --base, and a named or default repository weren't clear.

cmdutil: do not duplicate stdout by makefileobj() It made output order unpredictable because two separate buffers are flushed individually. Let's use a thin wrapper that just sends close() to black hole.

cmdutil: reimplement file wrapper that disables close() There's no need to dynamically create wrappedfileobj class and define close() as lambda. Also, __iter__() was missing.

export: remove useless comparison between file object and string It was introduced at 31aa2e5b0750, where "template" argument could be a file object. After that, a306837f8c87 added "len(template)", so "template" must be a string now. Therefore, "fp != template" should always be True. It seems 31aa2e5b0750 was intended to work around a bug in TortoiseHg, and I'm sure I've fixed it completely in TortoiseHg source. https://selenic.com/pipermail/mercurial-devel/2011-February/028467.html

export: do not print '<fdopen>' as an output filename Because makefileobj() duplicates or wraps stdout, "fp != sys.stdout" didn't work correctly. Python doc states that special file objects are named in the form '<...>', and absolute filenames should never start with '<', we can ignore names start with '<'. We can't test fp.fileno() because fp may be a command-server channel. https://docs.python.org/2.7/library/stdtypes.html#file.name In the test output, "exporting patch:" line is printed after patch content. This is caused by fdopen() and will be fixed by the subsequent patch.

commandserver: implement name() to clarify channel is not a plain file Because unknown attributes are delegated to the underlying file object, commandserver channels said they were '<stdout>' or '<stdin>' even though they weren't. This patch makes them say '<X-channel>'.

histedit: delete to drop The default behaviour to forbid this makes a lot of sense for novice users because it's safeguarding them from dangerous behavior but making it configurable will be apprieciated by power users in at least one big organization. It allows an user to look an histedit rules from declarative perspective and make the rules reflect the state after histedit. If we can move lines t move commits why can't we drop lines to drop commits? Let's put this behind config knob and inform users about this feature the very moment they are trying to use it so they can choose desired behaviour.

clonebundles: fix bundlespec typo

record: fix hunk handling to remember the current function

record: turn on showfunc Always try to give diff context when doing an interactive record

parsers: use PyTuple_Pack instead of manual list-filling Suggested by Yuya.

tags: mention --quiet switch in help (issue4920)

merge with stable

histedit: omit useless message from update (edit) specifically: 0 files updated, 0 files merged, 0 files removed, 0 files unresolved

histedit: omit useless message from update (_histedit) specifically: 0 files updated, 0 files merged, 0 files removed, 0 files unresolved

histedit: omit useless message from update (histeditaction) specifically: 0 files updated, 0 files merged, 0 files removed, 0 files unresolved

update: add quietempty flag to _showstats if called with quietempty=True, suppress: 0 files updated, 0 files merged, 0 files removed, 0 files unresolved

histedit: omit useless message from abort specifically: 0 files updated, 0 files merged, 0 files removed, 0 files unresolved

hg: add quietempty flag to _showstats if called with quietempty=True, suppress: 0 files updated, 0 files merged, 0 files removed, 0 files unresolved

patch: disable nobinary when HGPLAIN=1 The diff output without binaries is definitely great for interactive users - a binary patch is not meaningful for them. Although setting diff.nobinary flag can break the automation. Let's force full output for automation.

help: add missed last new line to "internals" topic Caught by test-gendoc.t.

dirstate: make delaywrite sleep until the next multiple of n seconds Rather than sleep for 2 seconds, we sleep until the next even-numbered second, which has the same effect, but makes tests faster. This removes test-largefiles-update as the long pole of the test suite.

dirstate: only invoke delaywrite if relevant This avoids a significant amount of sleeping in some of our longest tests.

dirstate: move delaywrite logic from write to _write This will allow us to be smarter in upcoming patches.

run-tests: show scheduling with --showchannels This gives one line of output per second with one column per -j level that allows analyzing test scheduling problems. First 24 seconds of output at -j 30 looks like this: 0 . 1 = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = s. 2 c c o c r l g r s s = c p = c h c a h c g c h c b c c l l c ss 3 h o b o e a e u u u c o a h o e o c g o l h g h u o = a o = s 4 e n s n b r n n b b m t g n l n l w n o e w e n n e r g i . 5 c t o = a g d - r r = m c w v p v . e v g c e c d v x g . m 6 k r l r s e o t e e b a h e e . e . b e . k b k l e t e . p 7 - i e e e f c e p p u n b b r . r . - r . - - - e r e f . o . 8 p b t v - i . s o o n d o d t . t . c t . c s = 2 t n i . r 9 y - e s c l . t - . d - m i - . - . o - . o y r - - s l . t 10 3 p - e h e . s s . l t b r s . s . m s . d m e f s i e . . 11 - e c t e s . . v . e e . . v . v . m v . e r n o v o s . . 12 c r h . c - . . n . 2 m . . n . n . a n . . e a r n n . . . 13 o f e . k u . . . . - p . . - . - . n - . . v m m - . . . . 14 m . c . - p . . . . e l . . s . m . d s . . . e a e . . . . 15 p . k . r d . . . . x a . . i . o . s o . . . - t n . . . . 16 a . h . e a . . . . c t . . n . v . . u . . . m . c . . . . 17 t . e . s t . . . . h e . . k . e . . r . . . e . o . . . . 18 . . a . t e . . . . a . . . . . . . . c . . . r . d . . . . 19 . . d . o . . . . . n . . . . . . . . e . . . g . i . . . . 20 . . s . r . . . . . g . . . . . . . . . . . . e . n . . . . 21 . . . . e . . . . . e . . . . . . . . . . . . 2 . g . . . . 22 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 . . . . . . . . . . . . . . . . . . . . . . . . . = . . . . ^C Test names read off vertically, beginning with '='. Idle time (not shown) appears as blank space.

tests: avoid duplicate install steps in test-run-tests At several seconds each, this is significantly slowing down the test.

run-tests: add more scheduling weight hints The scheduler would like to order test execution by expected run-time, but doesn't know much about how long a test will run. It thus uses test size as a proxy for run-time. By tweaking these weights we can keep CPUs more evenly busy and thus finish sooner. In particular, this change pushes the three currently longest-running tests closer to the beginning: test-largefiles-update.t test-run-tests.t test-gendoc.t As the largefiles test is currently the long pole of the test suite with higher -j factors, the sooner it's started, the sooner the tests can end. We also up the weight on some shorter but long-running tests that could have previously delayed completion with low -j factors by running very close to the end.

run-tests: report timeouts in a less alarming fashion Rather than report timed-out tests like this: ERROR: test-convert-svn-sink.t output changed ! ..simply put a 't' rather than a '.' in the stream.

ui: try to handle $$ more robustly in prompts (issue4970)

merge with stable

import: reorder help text Try to place key concepts early+together.

import: add word to help text

import: refactor exact flag

help: filter extension commands

test-help: tighten grep patterns Help should output section headings, but no debug commands

summary: add troubles list to the output of hg summary This patch adds troubles information to the output of hg summary. Example line displayed in hg summary: unstable: 1 changeset

wix: add missing template

tests: add coverage to ensure Wix tracks 'help' and 'templates' files This would have caught the problem fixed by 65d2538ac993. There are other *.wxs files that can be checked, but they appear to be more complicated. For example, locale.wxs has what appears to be foreach loop support, as well as variable substitution. By checking `hg files` to determine tracked file, this is able to avoid false failures when other junk is present in the filesystem, like *.orig files. I can't tell if the map-cmdline.status file is not included on purpose, but I don't see the purpose of excluding it. The missing help files seem reasonable for Windows.

perf: adjust perfstartup() for Windows The /dev/null redirect was causing the following error: The system cannot find the path specified. Adjusting HGRCPATH as part of the command line causes the system to try to execute 'HGRCPATH'.

test-hgignore: conditionalize an illegal Windows filename

mac: fix percent-encoding of non-utf-8 characters (issue4999)

help: support loading sub-topics If a sub-topic/section is requested and the main topic corresponds to a topic with sub-topics, we now look for and return content for a sub-topic if found. With this patch, `hg help internals.X` now works. hgweb does not yet render sub-topics, however.

help: pass sub-topic into help query functions While we will likely only use this variable in helptopic(), all these functions are called with the same arguments, so we have to be consistent.

help: pass subtopic into help() Now that we have multiple directories where help topics can live, we need a mechanism to access them. We already use "." to separate topic from section. So it seems logical to also use "." to denote the sub-directory of a topic. This patch teaches the help command to parse out the possible sub-topic and pass it to the help system.