Added signature for changeset 158bdc896572

Added tag 3.7-rc for changeset 158bdc896572

shelve: lowercase flag description The help for --unknown is the *only* command that started with a capital letter

changegroup: cg3 has two empty groups *after* manifests changegroup.getchunks() determines the end of the stream by looking for an empty chunk group (two consecutive empty chunks). It ignores empty groups in the first two groups. Changegroup 3 introduced an empty chunk between the manifests and the files, which confuses getchunks(). Since it comes after the first two, getchunks() will stop there. Fix by rewriting getchunks so it first counts two groups (empty or not) and then keeps antostarts counting empty groups. With this counting, changegroup 1 and 2 have exactly one empty group after the first two groups, while changegroup 3 has two (one for directories and one for files). It's a little hard to test this at this point, but I have verified that this patch fixes narrowhg (which was broken before this patch). Also, future patches will fix "hg strip" with treemanifests, and once that's done, getchunks() will be tested through tests of "hg strip".

mq: check for reserved patch name with qimport -r (issue5033) Fix regression from 143b52fce68e. Catching aborts might not be pretty but it works and is a small change.

mq: refactor makepatchname into class method

repoview: fix corrupted hiddencache crash Mercurial (issue5042) Before this patch if the hiddencache existed but was empty, it would crash mercurial. This patch adds exception handling when reading the hiddencache to avoid the issue. When encountering a corrupted cache file we print a devel warning. There would be no point in issuing a normal warning as the user wouldn't be able to do anything about the situation. The warning looks like: devel-warn: corrupted hidden cache, removing it at: /path/to/repoview.py

repoview: add missing newline character in debug prints

commandserver: drop tell() and seek() from channels (issue5049) These operations are obviously invalid for file-like channels because they will read or write protocol headers. This patch works around the issue that "hg archive" generates a corrupted zip file on Windows commandserver because of unusable tell() implementation. But the problem still occurs without using a commandserver. $ hg archive -R not-small-repo -t zip - | cat > invalid.zip So, this patch cannot fix the issue5049 completely.

crecord: edit during hg crecord should preserve cursor position (issue5041) This patch adds a variable to keep track of what hunk was selected before the edit. We use that variable to select the hunk or its replacement after the edit.

templates: use canvaswidth instead of fixed width for canvas (issue2683) Before this patch, template files for "graph" web page use fixed width size "480" for canvas element. This causes pruned lanes and invisible vertexes, if there are 16 or more vertical lanes at once. In such case, part of graph in right side area over 480 is invisible, even though corresponded summary text blocks are visible correctly. This limitation isn't reasonable for workflow using many branches at once (e.g. "one branch per issue" workflow). There were changes below related to width of canvas: - 7359cb753a54 (templates: widen the graph canvas (issue2683)), released as a part of Mercurial 1.8.2 According to the description, this assumed that 15 parallel branches was enough for ordinary workflow, and bumped width of canvas up from 224 to 480. - d490edc71146 (hgweb: make graph data suitable for template usage), released as a part of Mercurial 2.3 This introduced "canvaswidth" template keyword as a part of refactoring around graph rendering. But 'width="480"' of canvas element in template files wasn't replaced by 'width="{canvaswidth}"' in it (or subsequent one). This patch uses dynamic value "{canvaswidth}" instead of fixed width size "480" for canvas element. This is posted for "stable", because: - this is re-fixing issue2683 - this is simple enough for stable - using "{canvaswidth}" doesn't require any additional cost Calculation of canvaswidth is already implied as a part of "graph" web command.

backout: fix --no-commit option (issue5054)

bundle: exit early when there are no commits to bundle Previously, if you passed a revset that resolved to no nodes, it would get interpreted by the changegroup discovery logic as 'bundle all my heads', which is not what the user asked. Let's exit early when we notice this case. It could be argued that the changeset discovery logic should be smarter and only assume 'all heads' if the incoming heads parameter is None, but that's a much riskier change.

zeroconf: access repo on hgweb_mod properly (issue5036) hgweb_mod.hgweb.repo disappeared in ae33fff17c1e (hg: establish a cache for localrepository instances) and the code for accessing repo instances from hgweb was later refactored to go through a cache-aware context manager. Adapt zeroconf to access the repo instance via the new API.

merge default into stable for 3.7 code freeze

shelve: permit shelves to contain unknown files If an emergency comes in while you're in the middle of an experimental change, it can be useful to shelve not just files hg already tracks but also your unknown files while you handle the emergency. This is especially true if you have hooks intended to prevent you from forgetting to add new code before you push. Teach "hg shelve" to optionally shelve unknown files, not just tracked files. This is functionally similar to --addremove, but with two differences: 1) Deleted files are not removed. 2) Files added during shelve creation are tracked in extra so that they can be forgotten by "hg unshelve". When unshelving, we take care to only forget files if they've been created during the unshelve operation; if you add a file that's being tracked in a shelve as an unknown file, it should not become unknown again when the shelve is unshelved.

localrepo: don't reference transaction from hook closure (issue5043) Before, the hook() closure (which is called as part of locking hooks) would maintain a reference to a transaction instance (which should be finalized by the time lock hooks are called). Because we accumulate hook() instances when there are multiple transactions per lock, this would result in holding references to the transaction instances which would lead to higher memory utilization. Creating a reference to the hook arguments dict minimizes the number of objects that are kept alive until the lock release hook runs, minimizing memory "leaks."

context: don't use util.cachefunc due to cycle creation (issue5043) util.cachefunc stores all arguments as the cache key. For filectxfn functions, the arguments include the memctx instance. This creates a cycle where memctx._filectxfn references self. This causes a memory leak. We break the cycle by implementing our own memoizing function that only uses the path as the cache key. Since each memctx has its own cache instance, there is no concern about invalid cache hits.

largefiles: actions will now always have a file - drop check

largefiles: make prompt order deterministic 42ae1b1f048f introduced iteration of a set. Make it stable.

largefiles: fix commit of missing largefiles 832c98d79587 improved merging of standin files referencing missing largefiles. It did however not test or fix commits of such merges; it would abort. To fix that, change copytostore to skip and warn about missing largefiles with a message similar the one for failing get from remote filestores. (It would perhaps in both cases be better to emit a more helpful warning like "warning: standin file for large1 references 58e24f733a which can't be found in the local store".) To test this, make sure commit doesn't find the "missing" largefile in the global usercache. For further testing, verify that update and status works as expected after this. This will also effectively backout 63116d47cc3f.

diff: don't crash when merged-in addition is copied Similar to what was explained in the previous commit, the diff code expected copy source to be in "ctx1", which is not always the case during a merge. This has been broken since before hg 2.0. Also similar to the previous commit, we fix the problem by fixing up the copy dict.

diff: don't crash when merged-in addition was removed (issue4786) During a merge, if the user removes a file that came from parent 2 and did not exist in parent 1, the file's status will be "removed". This surprises the diff code, which crashes because it expects removed files exist in parent 1. This has been broken since 377124ba6b10 (trydiff: use 'not in addedset' for symmetry with 'not in removedset', 2014-12-23). Fix by fixing up the list of removed file, similar to how we currently fix up the list of modified and added files during a merge.

diff: move status fixup earlier, out of _filepairs() This prepares for future patches, and it also lets us remove the ugly "ctx1" argument to _filepairs() (ugly because of its assymmetry -- there's no "ctx2" argument).

graft: warn when -r is combined with revisions as positional arguments The behaviour in this case is undefined. Instead of silently doing something "random" and surprising, at least issue a warning. (This should perhaps be considered a "deprecation" and turned into an error in a future release.)

graft: clarify in help that `-r` is not just optional Positional parameters are also treated as revisions, but the order of revisions matters and it will often be wrong if the user understands it as `-r` taking multiple revisions as `-r REV1 REV2`. (Alternatively, `-r` could be turned into a no-op flag as the documentation suggests. That would however be less "semantic markup" and I agree with the implementation in 55e7f352b1d3 but not the documentation.)

streamclone: use backgroundfilecloser (issue4889) Closing files that have been appended to is slow on Windows/NTFS. CloseHandle() calls on this platform often take 1-10ms - and that's on my i7-6700K Skylake processor with a modern and fast SSD. Contrast with other I/O operations, such as writing data, which take <100us. This means that creating/appending thousands of files can add significant overhead. For example, cloning mozilla-central creates ~232,000 revlog files. Assuming 1ms per CloseHandle(), that yields 232s (3:52) of wall time waiting for file closes! The impact of this overhead can be measured most directly when applying stream clone bundles. Applying these files is effectively uncompressing a tar archive (read: it's very fast). Using a RAM disk (read: no I/O wait), the difference in wall time for a `hg debugapplystreamclonebundle` for a ~1731 MB mozilla-central bundle between Windows and Linux from the same machine is drastic: Linux: ~12.8s (128MB/s) Windows: ~352.0s (4.7MB/s) Windows is ~27.5x slower. Yikes! After this patch: Linux: ~12.8s (128MB/s) Windows: ~102.1s (16.1MB/s) Windows is now ~3.4x faster. Unfortunately, it is still ~8x slower than Linux. Profiling reveals a few hot code paths that could likely be improved. But those are for other patches. This patch introduces test-clone-uncompressed.t because existing tests of `clone --uncompressed` are scattered about and adding a variation for background thread closing to e.g. test-http.t doesn't feel correct.

streamclone: indent code This will make the subsequent patch easier to read.

scmutil: support background file closing Closing files that have been appended to is relatively slow on Windows/NTFS. This makes several Mercurial operations slower on Windows. The workaround to this issue is conceptually simple: use multiple threads for I/O. Unfortunately, Python doesn't scale well to multiple threads because of the GIL. And, refactoring our code to use threads everywhere would be a huge undertaking. So, we decide to tackle this problem by starting small: establishing a thread pool for closing files. This patch establishes a mechanism for closing file handles on separate threads. The coordinator object is basically a queue of file handles to operate on and a thread pool consuming from the queue. When files are opened through the VFS layer, the caller can specify that delay closing is allowed. A proxy class for file handles has been added. We must use a proxy because it isn't possible to modify __class__ on built-in types. This adds some overhead. But as future patches will show, this overhead is cancelled out by the benefit of closing file handles on background threads.

templatekw: add {namespaces} keyword This provides a general-purpose interface to all custom namespaces. The {namespaces} keyword honors the definition order of namespaces as they are kept by sortdict.

templatekw: move shownames() helper to be sorted alphabetically I'll add shownamespaces(), which is similar to this function. I want to put them nearby.

templater: make get(dict, key) return a single value This is necessary to obtain a _hybrid object from a dict. If get() yields a value, it would be stringified. I see no benefit to make get() lazy, so this patch just changes "yield" to "return".

templater: make _hybrid not callable to avoid conflicting semantics In templater, a callable symbol exists for lazy evaluation, which should have f(**mapping) signature. On the other hand, _hybrid.__call__(), which was introduced by 0b241d7a8c62, generates mapping for each element. This patch renames _hybrid.__call__() to _hybrid.itermaps() so that a _hybrid object can be a value of a mapping dict. {namespaces % "{namespace}: {names % "{name }"}\n"} ~~~~~ a _hybrid object

backout: commit changeset by default (BC) Add --no-commit flag to prevent it. This should make the hg user experience a little better. Some discussion can be found here: http://markmail.org/message/7jm7ro2ias6hxywy

help: move Windows 9x information to appropriate place

shelve: move commitfunc closer to use site Supporting shelving of unknown files needs this code motion. No functional changes.

commands: document clone bundles hooks and rollback behavior The added content is inside a verbose container. I figure it makes sense to explicitly document behavior, including with the caveat it may change later. People can't say they weren't warned!

clonebundles: improve BUNDLESPEC documentation Before the existence of `hg debugbundle --spec`, the process for defining the BUNDLESPEC value in manifests was not very clear and not trivial to automate, especially in the case of stream clone bundles. This patch adds documentation to note the existence of `hg debugbundle --spec`. We drop the reference to stream clone requirements handling because it is now redundant with `hg debugbundle --spec`. While we are here, we further reinforce the importance of defining BUNDLESPEC.

commands: teach debugbundle to print bundle specification This seems like the most logical place to put this functionality. Test coverage over existing known bundle specs has been added.

commands: use context manager for opened bundle file

exchange: implement function for inferring bundle specification We don't currently have a mechanism for inferring bundle spec strings from bundle files. This patch adds one. This will eventually be used to make the producing of clone bundles manifests easier.

streamclone: extract code for reading header fields So it can be called from another consumer in a future patch.

encoding: escape U+007F (DEL) character in JSON RFC 7159 does not state that U+007F must be escaped, but it is widely considered a control character. As '\x7f' is invisible on a terminal, and Python's json.dumps() escapes '\x7f', let's do the same.

run-tests: skip threading for a single test (issue5040) This version backs out 50e621fe0362 and implements it in a more consistent manner.

cleanup: use modern @property/@foo.setter property specification We can use this now that we're 2.6+, and this is more idiomatic modern Python.

buildrpm: use bash shebang, since we use bash features in the script As suggested by Bryan O'Sullivan.

mac: ignore resource fork when checking file sizes Some evil evil awful tool adds resource forks to files it's comparing. Our Mac-specific code to do bulk stats was accidentally using "total size" which includes those forks in the file size, causing them to be reported as modified. This changes it to only care about the normal data size and thus agree with what Mercurial's expecting.

copies: fix detection of divergent directory renames If we move all the files out of one directory, but into two different directories, we should not consider it a directory rename. The detection of this case was broken.

with: use context manager for transaction in strip

with: use context manager for transaction in pushphase

with: use context manager for transaction in strip

with: use a context manager for transaction in strip

with: use context manager in rebuildfncache

with: use context manager for transaction in consumev1

with: use context manager in _histedit

with: use context manager in amend

with: use context manager for transaction in changegroup apply (This needs some line wrapping due to the additional indent level. -mpm)

with: use context manager for transaction in rebase

with: use context manager for transaction in qimport

with: use context manager for transaction in qfinish

with: use context manager for transaction in mercurial_sink

transaction: turn a transaction into a Python context manager This lets us greatly simply acquire/release cycles. If the block completes without raising an exception, the transaction is closed. Code pattern before: try: tr = repo.transaction('x') # zillions of lines of code tr.close() finally: tr.release() And after: with tr.transaction('x'): # ...

with: use context manager for lock in pushphase

with: use context manager in rebuildfncache again

with: use context manager in streamclone consumev1

with: use context manager in manifest

with: use context manager in rename

with: use context manager in resolve

with: use context manager in unbundle

with: use context manager in update

with: use context manager in bisect save_state

with: use context manager in merge update

with: use context manager in _markchanges

with: use context manager in maybeperformlegacystreamclone

with: use context manager in verify

with: use context manager in qrename

with: use context manager in qfinish

with: use context manager in localrepo recover

with: use context manager in streamclone generatev1

with: use context manager in subrepo storeclean

with: use context manager in subrepo _cachestorehash

with: use context manager in largefiles commit

with: use context manager in unshelveabort

with: use context manager for wlock in transplant

with: use context manager for wlock in shelve stripcmd

with: use context manager for lock in continue

with: use context manager for wlock in unshelve

with: use context manager for wlock in shelve deletecmd

with: use context manager for wlock in shelve cleanupcmd

with: use context manager for wlock in shelve createcmd

with: use context manager for lock in histedit cleanupnode

with: use context manager for lock in qimport

with: use context manager for wlock in qfold

with: use context manager for wlock in qrefresh

with: use context manager for wlock in qpop

with: use context manager for wlock in qpush

with: use context manager for wlock in qnew

with: use context manager for wlock in mergeupdate

with: use context manager for wlock in overridepurge

with: use context manager for wlock in cmdutilforget

with: use context manager for wlock in overriderevert

with: use context manager for wlock in removelargefiles

with: use context manager for wlock in addlargefiles

with: use context manager for wlock in updatelfiles

with: use context manager for wlock in kw_dorecord

with: use context manager for wlock in kw_copy

with: use context manager for wlock in kw_amend

with: use context manager for wlock in keyword demo

with: use context manager for wlock in _kwfwrite

with: use context manager for wlock in sign

with: use context manager for wlock in checklookup

with: use context manager for wlock in copy

with: use context manager for wlock in workingctx.undelete

with: use context manager for wlock in workingctx.forget

with: use context manager for wlock in workingctx.add

with: use context manager for wlock in graft

with: use context manager for wlock in debugsetparents

with: use context manager for wlock in debugrebuilddirstate

with: use context manager for wlock in copy

with: use context manager for wlock in branch