revset: forward ordering requirement to argument of present() present() is special in that it returns the argument set with no modification, so the ordering requirement should be forwarded. We could make present() fix the order like orset(), but that would be silly because we know the extra filtering cost is unnecessary.

crecord: delete commented line

manifest: move dirlog up to manifestrevlog This removes dirlog and its associated cache from manifest and puts it in manifestrevlog. The notion of there being sub-logs is specific to the revlog implementation, and therefore belongs on the revlog class. This patch will enable future patches to move the serialization logic for manifests onto manifestrevlog, which will allow us to move manifest.add onto manifestlog in a way that it just calls out to manifestrevlog for the serialization.

manifest: move revlog specific options from manifest to manifestrevlog The manifestv2 and treeondisk options are specific to how we serialize the manifest into revlogs, so let's move them onto the manifestrevlog class. This will allow us to add a manifestlog.add() function in a future diff that will rely on manifestrevlog to make decisions about how to serialize the given manifest to disk. We have to move a little bit of extra logic about the 'dir' as well, since it is used in conjunction with the treeondisk option to decide the revlog file name. It's probably good to move this down to the manifestrevlog class anyway, since it's specific to the revlog.

manifest: adds manifestctx.readfast This adds a copy of manifest.readfast to manifestctx.readfast and adds a consumer of it. It currently looks like duplicate code, but a future patch causes these functions to diverge as tree concepts are added to the tree version.

manifest: add manifestctx.readdelta() This adds an implementation of readdelta to the new manifestctx class and adds a couple consumers of it. This currently appears to have some duplicate code, but future patches cause this function to diverge when things like "shallow" are introduced.

merge with stable

rebase: make debug logging more consistent We emit some lines that mix revision numbers with nodeids, which makes little sense to me.

revset: fix order of nested '_(|int|hex)list' expression (BC) This fixes the order of 'x & (y + z)' where 'y' and 'z' are trivial, and the other uses of _list()-family functions. The original functions are renamed to '_ordered(|int|hex)list' to say clearly that they do not follow the subset ordering.

revset: fix order of nested 'or' expression (BC) This fixes the order of 'x & (y + z)' where 'y' and 'z' are not trivial. The follow-order 'or' operation is slower than the ordered operation if an input set is large: #0 #1 #2 #3 0) 0.002968 0.002980 0.002982 0.073042 1) 0.004513 0.004485 0.012029 0.075261 #0: 0:4000 & (0:1099 + 1000:2099 + 2000:3099) #1: 4000:0 & (0:1099 + 1000:2099 + 2000:3099) #2: 10000:0 & (0:1099 + 1000:2099 + 2000:3099) #3: file("path:hg") & (0:1099 + 1000:2099 + 2000:3099) I've tried another implementation, but which appeared to be slower than this version. ss = [getset(repo, fullreposet(repo), x) for x in xs] return subset.filter(lambda r: any(r in s for s in ss), cache=False)

revset: add 'takeorder' attribute to mark functions that need ordering flag Since most functions shouldn't need 'order' flag, it is passed only when explicitly required. This avoids large API breakage.

revset: pass around ordering flags to operations Some operations and functions will need them to fix ordering bugs.

revset: add stub to handle parentpost operation All operations will take 'order' flag, but p1() function won't.

revset: infer ordering flag to teach if operation should define/follow order New flag 'order' is the hint to determine if a function or operation can enforce its ordering requirement or take the ordering already defined. It will be used to fix a couple of ordering bugs, such as: a) 'x & (y | z)' disregards the order of 'x' (issue5100) b) 'x & y:z' is listed from 'y' to 'z' c) 'x & y' can be rewritten as 'y & x' if weight(x) > weight(y) (a) and (b) are bugs of the revset core. Before this, there was no way to tell if 'orset()' and 'rangeset()' can enforce its ordering. These bugs could be addressed by overriding __and__() of the initial set to take the ordering of the other set: class fullreposet: def __and__(self, other): # allow other to enforce its ordering return other but it would expose (c), which is a hidden bug of optimize(). So, in either ways, optimize() have to know the current ordering requirement. Otherwise, it couldn't rewrite expressions by weights with no output change, nor tell how a revset function or operation should order the entries. 'order' is tri-state. It starts with 'define', and shifts to 'follow' by 'x & y'. It changes back to 'define' on function call 'f(x)' or function-like operation 'x (f) y' because 'f' may have its own ordering requirement for 'x' and 'y'. The state 'any' will allow us to avoid extra cost that would be necessary to constrain ordering where it isn't important, 'not x'.

revset: wrap arguments of 'or' by 'list' node This makes the number of 'or' arguments deterministic so we can attach additional ordering flag to all operator nodes. See the next patch. We rewrite the tree immediately after chained 'or' operations are flattened by simplifyinfixops(), so we don't need to care if arguments are stored in x[1] or x[1:].

journal: properly check for held lock (issue5349) The 'jlock' code meant to check for a held lock, but it actually just checking for a lock object. With CPython, this worked because the 'jlock' object is not referenced outside the '_write' function so reference counting would garbage collect it and the '_lockref' would return None. With pypy, the garbage collection would happen at an undefined time and the '_lockref' can still point to a 'jlock' object outside of '_write'. The right thing to do here is not only to check for a lock object but also to check if the lock is held. We update the code to do so and reuse a utility method that exist on 'localrepo' to help readability. This fix journal related tests with pypy.

ssl: handle a difference in SSLError with pypy (issue5348) The SSLError exception is a bit different with pypy (message is the first argument, not the second) This led the certificate error handling to crash when trying to extract the ssl error message. We now handle this different and 'test-https.t' is green again.

manifest: change manifestctx to not inherit from manifestdict If manifestctx inherits from manifestdict, it requires some weird logic to lazily load the dict if a piece of information is asked for. This ended up being complicated and unintuitive to use. Let's move the dict creation to .read(). This will make even more sense once we start adding readdelta() and other similar methods to manifestctx.

manifest: make one use of _mancache avoid manifestctxs In a future patch we will change manifestctx and treemanifestctx to no longer derive from manifestdict and treemanifest, respectively. This means that consumers of the _mancache will now need to be aware of the different between the two, until we get rid of the manifest entirely and the _mancache becomes only filled with ctxs. This fixes one case of it that can be fixed by using the other cache. Future patches will address the others uses using the upcoming manifestctx.read() function.

debugrevspec: add option to verify optimized result This provides a convenient way to diff "hg debugrevspec" outputs generated with/without --no-optimized option.

debugrevspec: add option to skip optimize() and evaluate unoptimized tree This will help debugging optimizer bugs. Maybe '--no-optimized' can be changed to '--optimized' (default: True) when flags series landed.

revset: remove showwarning option from expandaliases() Now all callers pass showwarning=ui.warn, so we no longer need the option to suppress warnings.

debugrevspec: evaluate tree built by itself Prepares for new option to evaluate an unoptimized tree. Since a revset expression is no longer parsed twice, alias warnings should be displayed at the first parsing stages. That's why showwarning=ui.warn is added.

localrepo: make _refreshfilecachestats unfiltered method to refresh correctly Before this patch, if transaction is started via "filtered repo" object, _refreshfilecachestats() at closing transaction doesn't refresh file stat of any @filecache properties correctly, because: - _refreshfilecachestats() omits refreshing file stat of a @filecache property, if it doesn't appear in self.__dict__ - if transaction is started via "filtered repo", _refreshfilecachestats() is applied on "filtered repo" because repo.transaction() adds "self._refreshfilecachestats" to post close procedures. repo.transaction() isn't unfiltered method, and "self" in it means "filtered repo" in this case. Transactions started by explicit repo.transaction() easily causes this situation. - _refreshfilecachestats() applied on "filtered repo" omits whole refreshing because @filecache properties are stored into "unfiltered repo", and appear only in self.__dict__ of "unfiltered repo". This incorrect refreshing causes unnecessary reloading from files. To refresh file stat of @filecache properties at closing transaction correctly, this patch makes _refreshfilecachestats() unfiltered method. This patch chooses making _refreshfilecachestats() unfiltered method instead of making transaction() unfiltered method, to reduce unexpected side effect.

streamclone: clear caches after writing changes into files for visibility Before this patch, streamclone-ed changes are invisible via @filecache properties to in-process procedures before closing transaction (e.g. pretxnclose python hook), if corresponded property is cached before consumev1(). Strictly speaking, caching should occur inside (store) lock for transaction. repo.invalidate() after closing transaction is too late to force @filecache properties to be reloaded from changed files at next access. For visibility of streamclone-ed changes to in-process procedures before closing transaction, this patch clears caches just after writing changes into files. BTW, regardless of changing in this patch, clearing cached properties in consumev1() causes inconsistency, if (1) transaction is started and (2) any @filecache property is changed before consumev1(). This patch also adds the comment to fix this (potential) inconsistency in the future.

localrepo: make invalidate avoid invalidating store inside transaction (API) Before this patch, invalidate() discards in-memory fncache changes, even inside transaction scope. Such changes should be written out at closing transaction. Otherwise, fncache might overlook newly added files. A file overlooked by fncache isn't accessible via store vfs, even if it actually exists in store. On the other hand, a non-existing file in fncache is less harmful, because fncachestore always examines whether a file actually exists or not before access. Therefore, discarding in-memory changes can be safely omitted. It is typical case that repo.invalidate() in streamclone is executed inside nested transaction. This patch makes invalidate() avoid invalidating store inside transaction. This patch focuses on describing only how invalidate() changes own behavior according to activity of transaction. Describing other detail of invalidate() in docstr will be done in another series, which refactors invalidate*() functions.

streamclone: force @filecache properties to be reloaded from file Before this patch, consumev1() invokes repo.invalidate() after closing transaction, to force @filecache properties to be reloaded from files at next access, because streamclone writes data into files directly. But this doesn't work as expected in the case below: 1. at closing transaction, repo._refreshfilecachestats() refreshes file stat of each @filecache properties with streamclone-ed files This means that in-memory properties are treated as valid. 2. but streamclone doesn't changes in-memory properties This means that in-memory properties are actually invalid. 3. repo.invalidate() just forces to examine file stat of @filecache properties at the first access after it Such examination should concludes that reloading from file isn't needed, because file stat was already refreshed at (1). Therefore, invalid in-memory cached properties (2) are unintentionally treated as valid (1). This patch invokes repo.invalidate() with clearfilecache=True, to force @filecache properties to be reloaded from file at next access. BTW, it is accidental that repo.invalidate() without clearfilecache=True in streamclone case seems to work as expected before this patch. If transaction is started via "filtered repo" object, repo._refreshfilecachestats() tries to refresh file stat of each @filecache properties on "filtered repo" object, even though all of them are stored into "unfiltered repo" object. In this case, repo._refreshfilecachestats() does nothing unintentionally, but this unexpected behavior causes reloading @filecache properties after repo.invalidate(). This is reason why this patch should be applied before making _refreshfilecachestats() correctly refresh file stat of @filecache properties.

manifest: backed out changeset bb3281b3fcaa There is some suspicious failure in evolution tests. This changeset was supposed to be dropped until we investigate.

manifest: backed out changeset b60a5fe98b73 There is some suspicious failure in evolution tests. This changeset was supposed to be dropped until we investigate.

debugrevspec: deprecate --optimize option This option has been superseded by '--show-stage NAME', and will cause confusion in future patches.

debugrevspec: add option to print parsed tree at given stages "-p <stage>" is useful for investigating parsing stages. With -p option, a transformed tree is printed no matter if it is changed or not, which allows us to know valid stage names by "-p all".

debugrevspec: transform and print parsed tree by stages Prepares for adding new option to print transformed tree at each stage.

debugrevspec: build parsed tree even if no --verbose specified Prepares for evaluating unoptimized tree. The output is suppressed by ui.note() anyway.

osutil: fix the declaration to work on different sizes of off_t

manifest: change manifestctx to not inherit from manifestdict If manifestctx inherits from manifestdict, it requires some weird logic to lazily load the dict if a piece of information is asked for. This ended up being complicated and unintuitive to use. Let's move the dict creation to .read(). This will make even more sense once we start adding readdelta() and other similar methods to manifestctx.

manifest: make uses of _mancache aware of contexts In a future patch we will change manifestctx and treemanifestctx to no longer derive from manifestdict and treemanifest, respectively. This means that consumers of the _mancache will now need to be aware of the different between the two, until we get rid of the manifest entirely and the _mancache becomes only filled with ctxs.

manifest: add treemanifestctx class Before we start using repo.manifestlog in the rest of the code base, we need to make sure that it's capable of returning treemanifests. As we add new functionality to manifestctx, we'll add it to treemanifestctx at the same time. We also comment out the manifestctx p1, p2, and linkrev fields for now, since we're not implementing them on treemanifest yet.

revset: add public function to create matcher from evaluatable tree "hg debugrevspec" will use it to evaluate unoptimized tree.

revset: make analyze() a separate step from optimize() This will allow us to evaluate unoptimized tree and compare the result with optimized one. The private _analyze() function isn't renamed since I'll add more parameters to it.

revset: extract tree transformation from optimize() This patch separates the simple tree transformation from the optimization step, which is called as _analyze() since I'll extend this function to infer ordering flags. I want to avoid making _optimize() more complicated. This will also allow us to evaluate unoptimized tree.

hghave: fix has_bzr() to not try to import RevisionSpec as module Since RevisionSpec isn't a module, "import bzrlib.revisionspec.RevisionSpec" would always raise ImportError.

update: label bookmark name in message We label bookmark name as such in various messages. This will help them to standout (or at least give the user the option to make them stand out). We use a distinct label for the 'active' bookmark, this can help users to catch bookmark operation affecting their working copy.

discovery: explicitly check for None in outgoing init f09d0004481c introduced default params for discovery.outgoing(), but it used a falsy check instead of an explicit check for None. The result is that callers that passed in an empty list would have that list overridden by the defaults, which is not the expected behavior. This was discovered by changes to the test-pushrebase.t test in Facebook's repository of mercurial extensions.

import: report directory-relative paths in error messages (issue5224) Import uses paths relative to the root of the repository, so when user imports patch with paths relative to the current working directory import aborts with 'unable to find file for patching'. This patch improves this situation by warning the user that paths are relative to the root of repository when patching fails.

commands: remove unecessary copying of list in graft()

revset: do not partial-match operator and function names in optimize() It was error-prone, and actually there was a typo, s/ancestorspec/ancestor/.

revset: remove false condition to process 'negate' operator 'negate' is mapped to 'string' at the above clause.

revset: make optimize() reject unknown operators This should have caught the bug of 'keyvalue' operator fixed at 5004ef47f437. The catch-all pattern is useless since optimize() should be aware of all known operators.

help: show content for explicitly disabled extension (issue5228)

obsolete: fix minor module documentation issues

debugextensions: hide "ships-with-hg-core" magic string (BC) Since we have "bundled" flag, we no longer need to rephrase "ships-with-hg-core" to show as "internal".

debugextensions: show ships-with-hg-core state as a separate field This is less magical than rephrasing ships-with-hg-core as internal, and we can distinguish "internal" liar. "tested with: internal" will be hidden by the next patch.

test-debugextensions: add dummy extension to make ifcontains() test more solid If testedwith were a string, ifcontains("3.2", testedwith) would match "3.2.1". New test added to cover that.

py3: remove use of *L syntax The int in Python 3 behaves as long so no need of L's in py3. Moreover we dont need long here.

util: rename checkcase() to fscasesensitive() (API) I always read the name "checkcase(path)" as "do we need to check for case folding at this path", but it's actually (I think) meant to be read "check if the file system cares about case at this path". I'm clearly not the only one confused by this as the dirstate has this property: def _checkcase(self): return not util.checkcase(self._join('.hg')) Maybe we should even inverse the function and call it fscasefolding() since that's what all callers care about?

manifest: call m1.load and m2.load before writing a subtree As part of refactoring the manifest, certain test cases started failing because writesubtrees was called with p1 and p2 manifests that had not been loaded (so accessing m1._dirs resulted in an empty set). Let's call _load on these before attempting to access _dirs. This was caught by tests when future patches were applied.

histedit: correct output of error when 'base' is from the edit list This was made more obvious by marmoute's recent rearrangement of this code.

py3: split check of pygments-using files from the rest of the tree If we don't do this, people without pygments installed in their Python 3 environment silently stop checking test-check-py3-compat, which isn't really what we wanted. This preserves stability of the test output while still letting anyone with a recent-enough Python 3 run the majority of the Python 3 compat checking test.

version: change "place" field of extension to "bundled" flag The name "place" sounds odd. We can simply expose raw boolean values instead of switching external/internal literals.

extensions: use ismoduleinternal() thoroughly "ships-with-hg-core" would be long enough to typo.

lock: show more detail for new-style locks in lock waiting message (issue4752) Improve the lock waiting warning message by explicitly saying that a host and process are holding the lock. This nudges confused new users in the direction of investigating the other process instead of removing the lock.

formatter: add context manager interface for convenience And port "hg files" to test it. As you can see, extra indent is necessary to port to this API. I don't think we should switch every fm.formatter() call to "with" statement.

histedit: remove now unused 'constraints' related code Now that the one action that need different logic handle that using inheritance and overriding, we can remove that code.

histedt: use inheritance to override the constraints in 'base' All actions but one actually have the same constraints when it comes to validate the 'action.node' value. So we actually just add this code to a method that can be overwritten in the one action where it matters. The now unused 'contraints' related enum and class attribute will be cleaned up in the next changeset.

histedit: move constraint verification to the 'action.verify' method Action has a method dedicated to verifying its validity. So we move code related to constrains into that method. This requires a bit more context to the 'verify' method in the same fashion we were passing the 'prev' argument. This is an extra step before we can simplify the constraint handling code further.

histedit: directly use node in 'verifyactions' It does not seem useful to convert to hex: it is an extra step and they are longer strings. So we stick to node for the logic. We only convert to short hex for error when needed. As a nice side effect this remove the explicit constant usage in'[12:]'. This will also help moving the code around later as we just have to access action.node.

histedit: temporarily shorten name of 'constraints' variable An upcoming changeset will make the line where this variable is used slightly too long. Other later changesets will clean that up further and makes the variable unnecessary, so this is only temporary and it does seems useful to put anything more complicate in place.

histedit: drop the 'nodetoverify' local variable We can just use 'action.node'.

histedit: make 'constraints' a class attribute instead of a method There does not seem to be a reason for this to be a method. So we initialise the class attribute once and for all at creation time and drop the instance method.

histedit: drop the 'nodetoverify' method That method is just returning self.node and is never overridden. We just use the attribute directly instead and get rid of the method. This is the beginning of series to simplify and unify verification of constraints for actions.

test-gpg: start gpg-agent by gpg-connect-agent only if GnuPG v2.1+ detected Danek Duvall found that gpg-connect-agent of GnuPG 2.0 never starts gpg-agent daemon. The 2.1 way is documented as "gpg-coonect-agent /bye" [1], which appears to be different from the 2.0 way [2]. [1]: https://www.gnupg.org/documentation/manuals/gnupg/Invoking-GPG_002dAGENT.html [2]: https://www.gnupg.org/documentation/manuals/gnupg-2.0/Invoking-GPG_002dAGENT.html Since "gpg-agent --daemon" of GnuPG 2.1 never prints environment variables, "eval $(gpg-agent --daemon)" would be valid only for GnuPG < 2.1, and we'll need a different workaround for 2.0. I have no 2.0 environment, I won't implement it.

rebase: properly calculate total commits to rebase (issue5347) Before this, predecessor commits were being included in the count

mpatch: add setup_mpatch_cffi.py This file got dropped when the corresponding setup.py changes were made.

journal: rename on disk files to 'namejournal' The 'journal' naming is already used by the transaction journal. Having an unrelated group of file with such a close naming is confusing and error prone. We rename the file used by the 'journal' extension to use 'namejournal' as the extension track the location of various 'names'.

tests: guard demandimport segment of test-extension.t

tests: skip demandimport if disabled demandimport and setuptools and decorator (from ironpython) and pygments leads to lots of fail. If demandimport is disabled we should skip testing it...

hghave: add demandimport checking

hghave: make bzr checks stricter My bzr does not have bzrlib.revisionspec.RevisionSpec, and thus tests were failing because convert refused to believe in bzr, but hghave without this change thought it was available.

help: document wire protocol commands

help: document wire protocol "handshake" protocol There isn't a formal handshake protocol in the wire protocol. But clients almost certainly need to perform particular actions before they can communicate with a server optimally. So document what that is so people understand what's going on at connection establishment time.

help: document wire protocol capabilities All capabilities from the history of the project are now documented.

crecord: properly handle files with No newline at eof (issue5268) Yes, this bug was a single character with the wrong case...

annotate: pre-calculate the "needed" dictionary (issue5360) The "needed" dict is used as a reference counter to free items in the giant "hist" dict. However, currently it is not very accurate and can lead to dropping "hist" items unnecessarily, for example, with the following DAG, -3- / \ 0--1--2--4-- The current algorithm will visit and calculate rev 1 twice, undesired. And it tries to be smart by clearing rev 1's parents: "pcache[1] = []" at the time hist[1] being accessed (note: hist[1] needs to be used twice, by rev 2 and rev 3). It can result in incorrect results if p1 of rev 4 deletes chunks belonging to rev 0. However, simply removing "needed" is not okay, because it will consume 10x memory: # without any change % HGRCPATH= lrun ./hg annotate mercurial/commands.py -r d130a38 3>&2 [1] MEMORY 49074176 CPUTIME 9.213 REALTIME 9.270 # with "needed" removed MEMORY 637673472 CPUTIME 8.164 REALTIME 8.249 This patch moves "needed" (and "pcache") calculation to a separate DFS to address the issue. It improves perf and fixes issue5360 by correctly reusing hist, while maintaining low memory usage. Some additional attempt has been made to further reduce memory usage, like changing "pcache[f] = []" to "del pcache[f]". Therefore the result can be both faster and lower memory usage: # with this patch applied MEMORY 47575040 CPUTIME 7.870 REALTIME 7.926 [1]: lrun is a lightweight sandbox built on Linux cgroup and namespace. It's used to measure CPU and memory usage here. Source code is available at github.com/quark-zju/lrun.

help: document wire protocol transport protocols The HTTP and SSH transport protocols are documented. This includes how commands and arguments are serialized as well as response types.

help: internals topic for wire protocol The Mercurial wire protocol is under-documented. This includes a lack of source docstrings and comments as well as pages on the official wiki. This patch adds the beginnings of "internals" documentation on the wire protocol. The documentation should have nearly complete coverage on the lower-level parts of the protocol, such as the different transport mechanims, how commands and arguments are sent, capabilities, and, of course, the commands themselves. As part of writing this documentation, I discovered a number of deficiencies in the protocol and bugs in the implementation. I've started sending patches for some of the issues. I hope to send a lot more. This patch starts with the scaffolding for a new internals page.

grep: add formatter support Several fields are renamed to be consistent with the annotate command, which doesn't mean the last call for the name unification [1]. Actually, I'd rather rename line_number to linenumber, linenum, lineno or line, but I want to port the grep command to formatter first. [1]: https://www.mercurial-scm.org/wiki/GenericTemplatingPlan#Dictionary I don't have any better name for the list of matched/unmatched texts, so they are just called as "texts".

grep: build list of all columns regardless of display options These columns should always be available in JSON or template outputs. The "change" column is excluded because it has no useful data unless --all is specified.

grep: build list of columns without "grep." label prefix Prepares for formatter support. We need field names without "grep.".

grep: factor out function that prints matched line with labels Prepares for formatter support.

grep: refactor loop that yields matched text with label As preparation for formatter support, this and the next patch split linestate.__iter__() into two functions, line scanner and displayer. New code uses regexp.search(str, pos) in place of regexp.search(substr), which appears to fix a bug of highlighting.

Added signature for changeset ccd436f7db6d

Added tag 3.9.1 for changeset ccd436f7db6d

bundle2: localize handleoutput remote prompts Code archaeology suggests that there was no good reason for this not to be localized. 'remote: ' is already localized elsewhere.

i18n-pt_BR: synchronized with a12d13eac513

files: change documentation to match its behaviour (issue5276) Documentation gave the usage pattern as '[OPTION]... [PATTERN]...' when the command match given files as relpaths by default.

templater: add template path to __base__ search This does a fall-back check for style files or directories that are in Mercurial's template path for user convenience. We intentionally don't use this for the built-in coal style because we don't want the style to mysteriously break if the working directory just happens to have a file named "paper".

bundle2: fail faster when interrupted Before this patch, bundle2 application attempted to consume remaining bundle2 part data when the process is interrupted (SIGINT) or when sys.exit is called (translated into a SystemExit exception). This meant that if one of these occurred when applying a say 1 GB changegroup bundle2 part being downloaded over a network, it may take Mercurial *several minutes* to terminate after a SIGINT because the process is waiting on the network to stream megabytes of data. This is not a great user experience and a regression from bundle1. Furthermore, many process supervisors tend to only give processes a finite amount of time to exit after delivering SIGINT: if processes take too long to self-terminate, a SIGKILL is issued and Mercurial has no opportunity to clean up. This would mean orphaned locks and transactions. Not good. This patch changes the bundle2 application behavior to fail faster when an interrupt or system exit is requested. It does so by not catching BaseException (which includes KeyboardInterrupt and SystemExit) and by explicitly checking for these conditions in yet another handler which would also seek to the end of the current bundle2 part on failure. The end result of this patch is that SIGINT is now reacted to significantly faster: the active transaction is rolled back immediately without waiting for incoming bundle2 data to be consumed. This restores the pre-bundle2 behavior and makes Mercurial treat signals with the urgency they deserve.

blackbox: also log alias expansions This should be extremely useful for helping users debug without having to see their complete configuration. Shell aliases do not get their expansion logged, because we don't look and see if we're in a repo before we dive into the execution of a shell alias. As a result, the ui object doesn't know where to log.

test-clonebundles: accept another error string here This is what happens if you run the test in a FreeBSD Jail, rather than "connection refused".

tests: guard test-archive-symlink with unziplinks check This fixes the test on FreeBSD, where the stock unzip doesn't understand symlinks.

hghave: add a check for unzip(1) that understands symlinks unzip(1) from the FreeBSD base system does not understand symlinks, so test-archive-symlinks is busted.

test-https: drop two spurious --traceback flags These make test failures scarier than they are.

extensions: change magic "shipped with hg" string I've caught multiple extensions in the wild lying about being 'internal', so it's time to move the goalposts on people. Goalpost moving will continue until third party extensions stop trying to defeat the system.

version: add formatter support The license message isn't exported, which I don't think is useful and I couldn't find a way to restructure it for JSON or template outputs.

version: factor out mapping of internal/external labels Prepares for formatter support, where translation should be disabled conditionally.

version: always build list of extension versions This patch just moves "if ui.verbose:" to the inner loop, as preparation for formatter support.

formatter: add fm.nested(field) to either write or build sub items We sometimes need to build nested items by formatter, but there was no convenient way other than building and putting them manually by fm.data(): exts = [] for n, v in extensions: fm.plain('%s %s\n' % (n, v)) exts.append({'name': n, 'ver': v}) fm.data(extensions=exts) This should work for simple cases, but doing this would make it harder to change the underlying data type for better templating support. So this patch provides fm.nested(field), which returns new nested formatter (or self if items aren't structured and just written to ui.) A nested formatter stores items which will later be rendered by the parent formatter. fn = fm.nested('extensions') for n, v in extensions: fn.startitem() fn.write('name ver', '%s %s\n', n, v) fn.end() Nested items are directly exported to a template for now: {extensions % "{name} {ver}\n"} There's no {extensions} nor {join(extensions, sep)} yet. I have a plan for them by extending fm.nested() API, but I want to revisit it after trying out this API in the real world.

formatter: factor out format*() functions to separate classes New converter classes will be reused by a nested formatter. See the next patch for details. This change is also good in that the default values are defined uniquely by the baseformatter.

crecord: restore SIGWINCH handler before return Previously, the SIGWINCH handler does not get cleared and if the commit message editor also needs SIGWINCH handling (like vim), the two SIGWINCH handlers (the editor's, ours) will have a race. And we may erase the editor's screen content. This patch restores SIGWINCH handler to address the above issue.

bdiff: implement cffi version of bdiff

bdiff: implement cffi version of blocks

util: checknlink should remove file it creates if an exception occurs There's no reason to leave the file behind.

merge: remove files with extra actions from merge action list See the comment for a detailed explanation why. Even though this is a bug, I've sent it to 'default' rather than 'stable' because it isn't triggered in any code paths in stock Mercurial, just with the merge driver included. For the same reason I haven't included any tests here -- the merge driver is getting a new test.

revlog: use an LRU cache for delta chain bases Profiling using statprof revealed a hotspot during changegroup application calculating delta chain bases on generaldelta repos. Essentially, revlog._addrevision() was performing a lot of redundant work tracing the delta chain as part of determining when the chain distance was acceptable. This was most pronounced when adding revisions to manifests, which can have delta chains thousands of revisions long. There was a delta chain base cache on revlogs before, but it only captured a single revision. This was acceptable before generaldelta, when _addrevision would build deltas from the previous revision and thus we'd pretty much guarantee a cache hit when resolving the delta chain base on a subsequent _addrevision call. However, it isn't suitable for generaldelta because parent revisions aren't necessarily the last processed revision. This patch converts the delta chain base cache to an LRU dict cache. The cache can hold multiple entries, so generaldelta repos have a higher chance of getting a cache hit. The impact of this change when processing changegroup additions is significant. On a generaldelta conversion of the "mozilla-unified" repo (which contains heads of the main Firefox repositories in chronological order - this means there are lots of transitions between heads in revlog order), this change has the following impact when performing an `hg unbundle` of an uncompressed bundle of the repo: before: 5:42 CPU time after: 4:34 CPU time Most of this time is saved when applying the changelog and manifest revlogs: before: 2:30 CPU time after: 1:17 CPU time That nearly a 50% reduction in CPU time applying changesets and manifests! Applying a gzipped bundle of the same repo (effectively simulating a `hg clone` over HTTP) showed a similar speedup: before: 5:53 CPU time after: 4:46 CPU time Wall time improvements were basically the same as CPU time. I didn't measure explicitly, but it feels like most of the time is saved when processing manifests. This makes sense, as large manifests tend to have very long delta chains and thus benefit the most from this cache. So, this change effectively makes changegroup application (which is used by `hg unbundle`, `hg clone`, `hg pull`, `hg unshelve`, and various other commands) significantly faster when delta chains are long (which can happen on repos with large numbers of files and thus large manifests). In theory, this change can result in more memory utilization. However, we're caching a dict of ints. At most we have 200 ints + Python object overhead per revlog. And, the cache is really only populated when performing read-heavy operations, such as adding changegroups or scanning an individual revlog. For memory bloat to be an issue, we'd need to scan/read several revisions from several revlogs all while having active references to several revlogs. I don't think there are many operations that do this, so I don't think memory bloat from the cache will be an issue.

revlog: remove unused variables

util: properly implement lrucachedict.get() Before, it was returning the raw _lrucachenode instance instead of its value.

manifest: change changectx to access manifest via manifestlog This is the first place where we'll start using manifestctx instances instead of manifestdict. This will facilitate using different manifestctx implementations in the future.

manifest: use property instead of field for manifest revlog storage The file caches we're using to avoid reloading the manifest from disk everytime has an annoying bug that causes the in memory structure to not be reloaded if the mtime and the size haven't changed. This causes a breakage in the tests because the manifestlog is not being reloaded after a commit+strip operation in mq (the mtime is the same because it all happens in the same second, and the resulting size is the same because we add 1 and remove 1). The only reason this doesn't affect the manifest itself is because we touch it so often that we had already reloaded it after the commit, but before the strip. Once the entire manifest has migrated to manifestlog, we can get rid of these properties, since then the manifestlog will be touched after the commit, but before the strip, as well.

manifest: introduce manifestlog and manifestctx classes This is the start of a large refactoring of the manifest class. It introduces the new manifestlog and manifestctx classes which will represent the collection of all manifests and individual instances, respectively. Future patches will begin to convert usages of repo.manifest to repo.manifestlog, adding the necessary functionality to manifestlog and instance as they are needed.

manifest: make manifest derive from manifestrevlog As part of our refactoring to split the manifest concept from its storage, we need to start moving the revlog specific parts of the manifest implementation to a new class. This patch creates manifestrevlog and moves the fulltextcache onto the base class.