revlog: drop silly `raw` parameter to `rawdata` function This is a leftover from `revision` and does not make sense for `rawdata`

perf: don't depend on pycompat for older Mercurial versions We already define local _sysstr() and _xrange() for compatibility, so we should use those. Also create a similar local _bytestr() corresponding to pycompat.bytestr(). Differential Revision: https://phab.mercurial-scm.org/D6745

perf: don't try to call `util.queue` on Mercurial version before it existed Differential Revision: https://phab.mercurial-scm.org/D6744

perf: handle NameError for `pycompat.foo` when pycompat wasn't imported On old Mercurial versions, we won't have a pycompat variable defined, and then `pycompat.foo` will raise a NameError. Differential Revision: https://phab.mercurial-scm.org/D6743

rawdata: update callers in shallowbundle We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in storageutils We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in delta utils We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in repository We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in testing/storage.py We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in test-revlog-raw We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in lfs' tests We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in lfs' wrapper We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update caller in wireprotov2server We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in debugcommands We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in sqlitestore We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update caller in remotefilelog We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in bundlerepo We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update callers in context We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

rawdata: update caller in revlog We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.

setup: fix a sorting issue I noticed in package names Differential Revision: https://phab.mercurial-scm.org/D6733

py3: do not convert rust module/attribute names to bytes policy.import*() functions expect system strings.

transplant: unnest --stop case It should be aligned with --continue.

rust-discovery: use while loop instead of match + break This looks slightly nicer.

rust-discovery: remove useless extern crate

transplant: added support for --stop flag This adds fuctionality for `--stop` flag to `transplant`. A new method `stop` is added to `transplanter` class containing logic to abort transplant. Tests are updated as shown. Differential Revision: https://phab.mercurial-scm.org/D6695

unshelve: abort on using --keep and --interactive together I am working on making interactive mode support `--keep` flag. Until we support the usage of `--interactive` and `--keep` together, let us abort on it. Differential Revision: https://phab.mercurial-scm.org/D6699

config: add experimental argument to the config registrar Until now, there are almost 28 config items which are considered as `experimental` but, not present in the `experimental` section of the registrar. This patch adds an `experimental` argument to the config registrar to mark such config items. Differential Revision: https://phab.mercurial-scm.org/D6728 Differential Revision: https://phab.mercurial-scm.org/D6746

tests: split joint repo/changelog fake into one for each type I'm just not comfortable with fakes that get overloaded for multiple types: too often it gets out of hand and becomes difficult to trace. Differential Revision: https://phab.mercurial-scm.org/D6725

fix: pass line ranges as value instead of callback The callback no longer takes any arguments from the inner function, so we might as well call it sooner and pass the value instead. Note the value still needs to be recomputed every iteration to account for the previous iteration's changes to the file content. Differential Revision: https://phab.mercurial-scm.org/D6727

fix: correctly parse the :metadata subconfig It's being handled as a string instead of a bool, though the thruthiness of the string makes the feature still essentially work. Added a regression test. Differential Revision: https://phab.mercurial-scm.org/D6726

fix: allow tools to use :linerange, but also run if a file is unchanged The definition of "unchanged" here is subtle, because pure deletion diff hunks are ignored. That means this is different from using the --whole flag. This change allows you to configure, for example, a code formatter that: 1. Formats specific line ranges if specified via flags 2. Does not format the entire file when there are no line ranges provided 3. Performs some other kind of formatting regardless of provided line ranges This sounds a little far fetched, but it is meant to address a specific corner case encountered in Google's use of the fix extension. The default behavior is kept because it exists to prevent mistakes that could erase uncommitted changes. Differential Revision: https://phab.mercurial-scm.org/D6723

rust-dirstate: call rust dirstatemap from Python Since Rust-backed Python classes cannot be used as baseclasses (for rust-cpython anyway), we use composition rather than inheritance. This also allows us to keep the IO operations in the Python side, removing (for now) the need to rewrite VFS in Rust, which would be a heavy undertaking. Differential Revision: https://phab.mercurial-scm.org/D6634

rust-dirstate: rust-cpython bridge for dirstatemap This change also showcases the limitations of the `py_shared_ref!` macro. See the previous commit 'rust-dirstate: rust implementation of dirstatemap` for an explanation for the TODOs in the code. Differential Revision: https://phab.mercurial-scm.org/D6633

rust-dirstate: rust implementation of dirstatemap The `dirstatemap` is one of the last building blocks needed to get to a `dirstate.walk` Rust implementation. Disclaimer: This change is part of a big (10) series of patches, all of which started as one big changeset that took a long time to write. This `dirstatemap` implementation is a compromise in terms of complexity both for me and for the reviewers. I chose to submit this patch right now because while it is not perfect, it works and is simple enough (IMHO) to be reviewed. The Python implementation uses a lot of lazy propertycaches, breaks encapsulation and is used as an iterator in a lot of places, all of which dictated the somewhat unidiomatic patterns in this change. Like written in the comments, rewriting this struct to use the typestate pattern might be a good idea, but this is a good first step. Differential Revision: https://phab.mercurial-scm.org/D6632

rust-cpython: add macro for sharing references Following an experiment done by Georges Racinet, we now have a working way of sharing references between Python and Rust. This is needed in many points of the codebase, for example every time we need to expose an iterator to a Rust-backed Python class. In a few words, references are (unsafely) marked as `'static` and coupled with manual reference counting; we are doing manual borrow-checking. This changes introduces two declarative macro to help reduce boilerplate. While it is better than not using macros, they are not perfect. They need to: - Integrate with the garbage collector for container types (not needed as of yet), as stated in the docstring - Allow for leaking multiple attributes at the same time - Inject the `py_shared_state` data attribute in `py_class`-generated structs - Automatically namespace the functions and attributes they generate For at least the last two points, we will need to write a procedural macro instead of a declarative one. While this reference-sharing mechanism is being ironed out I thought it best not to implement it yet. Lastly, and implementation detail renders our Rust-backed Python iterators too strict to be proper drop-in replacements, as will be illustrated in a future patch: if the data structure referenced by a non-depleted iterator is mutated, an `AlreadyBorrowed` exception is raised, whereas Python would allow it, only to raise a `RuntimeError` if `next` is called on said iterator. This will have to be addressed at some point. Differential Revision: https://phab.mercurial-scm.org/D6631

rust-docstrings: add missing module docstrings Differential Revision: https://phab.mercurial-scm.org/D6630

rust-dirstate: improve API of `DirsMultiset` - Use opaque `Iterator` type instead of implementation-specific one from `HashMap` - Make `DirsMultiset` behave like a set both in Rust and from Python Differential Revision: https://phab.mercurial-scm.org/D6690

rust-dirstate: use EntryState enum instead of literals This improves code readability quite a bit, while also adding a layer of safety because we're checking the state byte against the enum. Differential Revision: https://phab.mercurial-scm.org/D6629

rust-parsers: switch to parse/pack_dirstate to mutate-on-loop Both `parse_dirstate` and `pack_dirstate` can operate directly on the data they're passed, which prevents the creation of intermediate data structures, simplifies the function signatures and reduces boilerplate. They are exposed directly to the Python for now, but a later patch will make use of them inside `hg-core`. Differential Revision: https://phab.mercurial-scm.org/D6628

rust-parsers: move parser bindings to their own file and Python module This tidies up the Rust side while simplifying the Python side. Differential Revision: https://phab.mercurial-scm.org/D6627

rust-dirstate: create dirstate submodule in hg-cpython This module will soon hold multiple files, this change is to make the review process easier. Differential Revision: https://phab.mercurial-scm.org/D6626

rust-discovery: using from Python code As previously done in other topics, the Rust version is used if it's been built. The version fully in Rust of the partialdiscovery class has the performance advantage over the Python version (actually using the Rust MissingAncestor) if the undecided set is big enough. Otherwise no sampling occurs, and the discovery is reasonably fast anyway. Note: it's hard to predict the size of the initial undecided set, it can depend on the kind of topological changes between the local and remote graphs. The point of the Rust version is to make the bad cases acceptable. More specifically, the performance advantages are: - faster sampling, especially takefullsample() - much faster addmissings() in almost all cases (see commit message in grandparent of the present changeset) - no conversion cost of the undecided set at the interface between Rust and Python == Measurements with big undecided sets For an extreme example, discovery between mozilla-try and mozilla-unified (over one million undecided revisions, same case as in dbd0fcca6dfc), we get roughly a x2.5/x3 better performance: Growing sample size (5% starting with 200): time goes down from 210 to 72 seconds. Constant sample size of 200: time down from 1853 to 659 seconds. With a sample size computed from number of roots and heads of the undecided set (`respectsize` is `False`), here are perfdiscovery results: Before ! wall 9.358729 comb 9.360000 user 9.310000 sys 0.050000 (median of 50) After ! wall 3.793819 comb 3.790000 user 3.750000 sys 0.040000 (median of 50) In that later case, the sample sizes are routinely in the hundreds of thousands of revisions. While still faster, the Rust iteration in addmissings has less of an advantage than with smaller sample sizes, but one sees addcommons becoming faster, probably a consequence of not having to copy big sets back and forth. This example is not a goal in itself, but it showcases several different areas in which the process can become slow, due to different factors, and how this full Rust version can help. == Measurements with small undecided sets In cases the undecided set is small enough than no sampling occurs, the Rust version has a disadvantage at init if `targetheads` is really big (some time is lost in the translation to Rust data structures), and that is compensated by the faster `addmissings()`. On a private repository with over one million commits, we still get a minor improvement, of 6.8%: Before ! wall 0.593585 comb 0.590000 user 0.550000 sys 0.040000 (median of 50) After ! wall 0.553035 comb 0.550000 user 0.520000 sys 0.030000 (median of 50) What's interesting in that case is the first addinfo() at 180ms for Rust and 233ms for Python+C, mostly due to add_missings and the children cache computation being done in less than 0.2ms on the Rust side vs over 40ms on the Python side. The worst case we have on hand is with mozilla-try, prepared with discovery-helper.sh for 10 heads and depth 10, time goes up 2.2% on the median. In this case `targetheads` is really huge with 165842 server heads. Before ! wall 0.823884 comb 0.810000 user 0.790000 sys 0.020000 (median of 50) After ! wall 0.842607 comb 0.840000 user 0.800000 sys 0.040000 (median of 50) If that would be considered a problem, more adjustments can be made, which are prematurate at this stage: cooking special variants of methods of the inner MissingAncestors object, retrieving local heads directly from Rust to avoid the cost of conversion. Effort would probably be better spent at this point improving the surroundings if needed. Here's another data point with a smaller repository, pypy, where performance is almost identical Before ! wall 0.015121 comb 0.030000 user 0.020000 sys 0.010000 (median of 186) After ! wall 0.015009 comb 0.010000 user 0.010000 sys 0.000000 (median of 184) Differential Revision: https://phab.mercurial-scm.org/D6430

rust-discovery: optimization of add commons/missings for empty arguments These two cases have to be catched early for different reasons. In the case of add_missing_revisions, we don't want to trigger the computation of the undecided set (and the children cache) too early: the later the better. In the case of add_common_revisions, the inner `MissingAncestors` object wouldn't know that all ancestors of its bases have already been removed from the undecided. In principle, that would in itself be a lead for further improvement: this remove_ancestors_from could be more incremental, but the current performance seems to be good enough. Differential Revision: https://phab.mercurial-scm.org/D6429

rust-discovery: using the children cache in add_missing The DAG range computation often needs to get back to very old revisions, and turns out to be disproportionately long, given that the end goal is to remove the descendents of the given missing revisons from the undecided set. The fast iteration capabilities available in the Rust case make it possible to avoid the DAG range entirely, at the cost of precomputing the children cache, and to simply iterate on children of the given missing revisions. This is a case where staying on the same side of the interface between the two languages has clear benefits. On discoveries with initial undecided sets small enough to bypass sampling entirely, the total cost of computing the children cache and the subsequent iteration becomes better than the Python + C counterpart, which relies on reachableroots2. For example, on a repo with more than one million revisions with an initial undecided set of 11 elements, we get these figures: Rust version with simple iteration addcommons: 57.287us first undecided computation: 184.278334ms first children cache computation: 131.056us addmissings iteration: 42.766us first addinfo total: 185.24 ms Python + C version first addcommons: 0.29 ms addcommons 0.21 ms first undecided computation 191.35 ms addmissings 45.75 ms first addinfo total: 237.77 ms On discoveries with large undecided sets, the initial price paid makes the first addinfo slower than the Python + C version, but that's more than compensated by the gain in sampling and subsequent iterations. Here's an extreme example with an undecided set of a million revisions: Rust version: first undecided computation: 293.842629ms first children cache computation: 407.911297ms addmissings iteration: 34.312869ms first addinfo total: 776.02 ms taking initial sample query 2: sampling time: 1318.38 ms query 2; still undecided: 1005013, sample size is: 200 addmissings: 143.062us Python + C version: first undecided computation 298.13 ms addmissings 80.13 ms first addinfo total: 399.62 ms taking initial sample query 2: sampling time: 3957.23 ms query 2; still undecided: 1005013, sample size is: 200 addmissings 52.88 ms Differential Revision: https://phab.mercurial-scm.org/D6428

discovery: new devel.discovery.randomize option By default, this is True, but setting it to False is a uniform way to kill all randomness in integration tests such as test-setdiscovery.t By "uniform" we mean that it can be passed to implementations in other languages, for which the monkey-patching of random.sample would be irrelevant. In the above mentioned test file, we use it right away, replacing the adhoc extension that had the same purpose, and to derandomize a case with many round-trips, that we'll need to behave uniformly in the Rust version. Differential Revision: https://phab.mercurial-scm.org/D6427

rust-discovery: optionally don't randomize at all, for tests As seen from Python, this is a new `randomize` kwarg in init of the discovery object. It replaces random picking by some arbitrary yet deterministic strategy. This is the same as what test-setdiscovery.t does, with the added benefit to be usable both in Python and Rust implementations. Differential Revision: https://phab.mercurial-scm.org/D6426

rust-discovery: exposing sampling to python Differential Revision: https://phab.mercurial-scm.org/D6425

tests: use `tr -d` and not `tr --delete` as the latter is absent on BSD tr(1) Differential Revision: https://phab.mercurial-scm.org/D6729

rust-discovery: takefullsample() core implementation take_full_sample() browses the undecided set in both directions: from its roots as well as from its heads. Following what's done on the Python side, we alter update_sample() signature to take a closure returning an iterator: either ParentsIterator or an iterator over the children found in `children_cache`. These constructs should probably be split off in a separate module. This is a first concrete example where a more abstract graph notion (probably a trait) would be useful, as this is nothing but an operation on the reversed DAG. A similar motivation in the context of the discovery process would be to replace the call to dagops::range in `add_missing_revisions()` with a simple iteration over descendents, again an operation on the reversed graph. Differential Revision: https://phab.mercurial-scm.org/D6424

rust-discovery: core implementation for take_quick_sample() This makes in particular `rand` no longer a testing dependency. We keep a seedable random generator on the `PartialDiscovery` object itself, to avoid lengthy initialization. In take_quick_sample() itself, we had to avoid keeping the reference to `self.undecided` to cope with the mutable reference introduced by the the call to `limit_sample`, but it's still manageable without resorting to inner mutability. Sampling being prone to be improved in the mid-term future, testing is minimal, amounting to checking which code path got executed. Differential Revision: https://phab.mercurial-scm.org/D6423

rust-discovery: read the index from a repo passed at init This makes the API of the Rust PartialDiscovery object now the same (or rather a subset) of the Python object, hence easier to control through module policy down the road. Differential Revision: https://phab.mercurial-scm.org/D6517

rust-discovery: accept the new 'respectsize' init arg At this stage, we don't do anything about it: it will be meaningful in sampling methods that aren't implemented yet. Differential Revision: https://phab.mercurial-scm.org/D6516

merge with stable

unshelve: forget unknown files after a partial unshelve This is a follow-up patch to 6957f7b93e03. This allows hg to forget unknown files after a partial unshelve. Differential Revision: https://phab.mercurial-scm.org/D6724

flagutil: move addflagprocessor to the new module (API)

flagutil: move insertflagprocessor to the new module (API)

flagutil: move REVIDX_KNOWN_FLAGS source of truth in flagutil (API) Since REVIDX_KNOWN_FLAGS is "not really a constant" (extension can update it) and python integer,... it needs to be the responsability of a single module and always accessed through the module. We update all the user and move the source of truth in flagutil.

flagutil: move the `flagprocessors` mapping in the new module This module is meant to host most of the flag processing logic. We start with the mapping between flag and processors.

flagutil: create a `mercurial.revlogutils.flagutil` module The flagprocessings logic is duplicated in 2 extra places, and usually in a less robust flavor. This is a maintenance nightmare that I would like to see cleaned up. To do so I am creating a `flagutil` module to move flag processings related code and make it easily reusable by other code.

rawdata: register the method for `ifiledata` The interface have a `revision(..., raw=False)` method so it should get a `rawdata` one. I am not sure why nothing complained about the lack of it earlier.

rawdata: implement the method for `unionrepo` too This is required for all implementations.

rawdata: implement the method for `remotefilelog` too This is needed for all storage implementations.

rawdata: implement `rawdata` for `simplestore` too This is needed for all implementation.

rawdata: forward `rawdata` call on `manifestlog` This needs to be sent to the underlying `revlog` too.

rawdata: implement `rawdata` for `sqlitestore` too This is a different store, it needs it declared.

rawdata: add the method to bundlerevlog The bundlerepo logic has its own `revision` method on its own `revlog` object. We need to "implement" `rawdata` there too.

rawdata: forward the method call on `filelog` object We have a new method, we need to expose it.

rawdata: introduce a `rawdata` method on revlog This method aims at replacing `revision(..., raw=True)` call. The purpose of data returned without and without raw are different enough that having two different method would make sense. This split is motivated by other work aiming at storing data on the side of the main revision of a revlog. Having a cleaner API makes it simpler to add this work. The series following this first changesets is organised as follow: 1) add `rawdata` method everywhere it is useful 2) update all caller 3) implement all `rawdata` method without using `revision` 4) deprecate the `rawdata` parameter of `revision`

revlog: split a `_revisiondata` method to file `revision` job We are about to introduce more public method to access revision data (eg: `rawdata`). revset subclass tend to recursively call `revision` which will create all kind of issue with the coming series. To avoid them we introduce an explicit difference between the internal call and the public all. This will be useful for later work anyway (so the subclass issue is just moving it earlier in the series). I am not sure if the subclass are actually doing something sensible. However, I am certain I don't want to be rabbit holed into figuring it out right now.

continue: added support for transplant This creates a seperate function `continuetransplant()` containing logic for resuming transplant from interrupted state. `continuetransplant()` is then registered as `continuefunc` for state detection API. Results are shown in tests. Differential Revision: https://phab.mercurial-scm.org/D6689

merge with stable

fncache: make debugrebuildfncache not fail on broken fncache The code reading the fncache changed in 5.0, to complain if the file is not \n terminated. This makes apparent the fact that the fncache gets corrupted. Make it possible to recover, instead of having `hg debugrebuildfncache` failing by saying `(run hg debugrebuildfncache)`. The corruption itself is most likely due to hg not using fsync in general, and so various bad things can happen. Here, the reported problems happened when running out of disk space. So I suspect that because the fncache is much bigger than the average commit/pull, when running out of disk space, the bulk of the pull may succeed, but the new fncache may get half-written and still renamed into place. Differential Revision: https://phab.mercurial-scm.org/D6722

fncache: show that debugrebuildfncache is partly broken Differential Revision: https://phab.mercurial-scm.org/D6721

test: further fixes to matching for run-tests.py bug The fix in bac24a8a095a did not fix the full issue, because the extra number also eat some of the separator space. Since we are already matching arbitrary number of space, we easily fix the matching.

branchmap: explicitly warm+write all subsets of the branchmap caches 'full' claims it will warm all of the caches that are known about, but this was not the case - it did not actually warm the branchmap caches for subsets that we haven't requested, or for subsets that are still considered "valid". By explicitly writing them to disk, we can force the subsets for ex: "served" to be written ("immutable" and "base"), making it cheaper to calculate "served" the next time it needs to be updated. Differential Revision: https://phab.mercurial-scm.org/D6710

changectx: extract explicit computechangesetfilesremoved method from context Right now, the logic around changeset centric removed files data are buried into the "changectx" code. We extract this code in a dedicated method (in the scmutil module) for clarity. This clarity will help to explicitly compute and caches these data in the future.

changectx: extract explicit computechangesetfilesadded method from context Right now, the logic around changeset centric added files data are buried into the "changectx" code. We extract this code in a dedicated method (in the scmutil module) for clarity. This clarity will help to explicitly compute and caches these data in the future.

demandimport: explicitly declare `_session` at the module level The `_session` module level variable is set within a function using the `global` keyword. This confuses my `test-check-pyflakes.t`. Explicitly declaring the variable at the top level solves the issue (and seems absolutely reasonable).

tests: give more room for slowness in test-run-tests.t The test expected any run-test.py run to end in less than 10 seconds. On slower loaded CI machine, this gets slower than that. We give a bit more room to the regexp.

copies: extract an explicit `computechangesetcopie` method from context Right now, the logic around changeset centric copies data are buried into the "changectx" code. We extract this code in a dedicated method (in the copies module) for clarity. This clarity will help to explicitly compute and caches these data in the future.

config: fix fm.data() handling of defaultvalue This is a follow-up patch to rHG51a2e3102db2. This moves `fm.data()` out of the if block in `commands.config()`. Differential Revision: https://phab.mercurial-scm.org/D6720

config: remove pycompat.bytestr() for defaultvalue This is a follow-up patch to 51a2e3102db2. This removes `pycompat.bytestr` to preserve `None` in `commands.config()`. Differential Revision: https://phab.mercurial-scm.org/D6712

unshelve: clear shelvedstate and _finishunshelve() on partial unshelve On a partial unshelve, `shelvedstate` was not cleared and `_finishunshelve()` was not called. Ideally, these should be called on this case. This patch makes `shelvedstate` to delete after a successful partial unshelve and calls `_finishunshelve()` in the same case. Differential Revision: https://phab.mercurial-scm.org/D6708

unshelve: delete shelvedstate after a successful unshelve --continue `unshelve --continue` was preventing the deletion of `shelvedstate` on a partial `unshelve`. Ideally, `shelvedstate` should be deleted after a successful `unshelve`. Now, the behavior of `unshelve --continue` will be as follows in interactive mode: 1] The user tried to `unshelve` changes interactively but, ran into conflicts. 2] They resolved the conflicts and triggered `unshelve --continue` but, unshelved changes partially. 3] Now, on trying to do `unshelve --continue` again will abort as the last `unshelve` was successful and we are deleting the `shelvedstate`. 4] If they want to unshelve the remaining shelved change, they need to trigger `unshelve` without `--continue`. Differential Revision: https://phab.mercurial-scm.org/D6694

unshelve: handle stripping changesets on interactive mode On interactive mode, changesets on `nodestoremove` should be stripped regardless of the shelve is partial or not. This patch modifies `unshelvecontinue()` to do that. Differential Revision: https://phab.mercurial-scm.org/D6686

byteify-strings: add --version argument This is indispensable for automated tools to detect changes in behavior.

byteify-strings: add space in special comments to silence flake8 error This is done soon enough that nobody has had the time to use this feature yet.

revset: drop argument when it's None getstack's definition is `getstack(repo, rev=None)`, so providing None explicitly is unnecessary. Moreover, when x is not None, it's definitely not a revision but a part of a parsed tree of revset arguments. Differential Revision: https://phab.mercurial-scm.org/D6707

stack: remove unnecessary reverse() predicate Stack already sorts revisions, so no need to do it twice. This change was a part of D2400, which didn't land for other reasons. See also D2399, where this change was suggested. Differential Revision: https://phab.mercurial-scm.org/D6706

automation: increase root volume size on Linux It is close to full in the AMI. I actually ran out of space running tests without increasing the volume size! Differential Revision: https://phab.mercurial-scm.org/D6716

automation: install Rust in Linux environment This will install Rust 1.31.1, 1.34.2, and whatever stable is at the time the install runs. We install 1.31.1 as our minimum supported Rust version (I think that's what we're currently targeting) and 1.34 because that's what Debian 10 is shipping. Differential Revision: https://phab.mercurial-scm.org/D6715

automation: update packages in requirements files We like keeping up to date. The content of the autogenerated files changed slightly because I used a newer version of `pip-compile` than what was used previously. Differential Revision: https://phab.mercurial-scm.org/D6714

automation: install latest Python versions This required bumping the pyenv commit so the new versions are available. Differential Revision: https://phab.mercurial-scm.org/D6713

upgrade: introduce the internal code for revlog cloning selection For now we still clone every single revlogs but all the selection mechanism is now in place in the lower layer. The next changesets will introduce the user interface part of the selection.

relnotes: copy "next" to "5.1" and clear "next" To avoid merge conflicts, we want to avoid modifying the file on multiple branches in parallel. This patch is therefore meant to be applied to the stable branch and then quickly be merged to default (at least before edits are made to relnotes/next there). Another option would have been to copy the file on the stable branch and to clear it on the default branch. However, that still results in conflicts if the copy is edited on the stable branch (Mercurial would try to apply the changes from the default branch to it). We could also delete the file in one commit and recreate it in another commit. However, Mercurial is quite inconsistent in what it considers a break in history (see test-copies-unrelated.t), so I'd like to avoid that. Differential Revision: https://phab.mercurial-scm.org/D6705

upgrade: introduce a _copyrevlog method This function copies a revlog from the old store to the new one, without re-adding all deltas manually. This will eventually save a lot of time when some revlog does not needs any conversions. Code actually using this will be introduced in later changesets.

upgrade: rename `_copyrevlogs` to `_clonerevlogs` The underlying revlog method is named `clone`, keeping the naming consistent seems clearer. This is motivated to clarify the difference with an (upcoming) function that simply copy revlog files as is.

upgrade: walk the source store file only once I don't expect this to have a significant performance impact, but it seems simpler and saner to do the operation only once and to keep the result around.

upgrade: always use full text if "full-add" mode is enable We should not be using a delta since the goal is to perform a full addition from scratch in all cases. Without this patch, `hg debugupgraderepo --optimize re-delta-fulladd --run` can crash.

byteify-strings: fix misalignment with multi-line parenthesis This improves the current fix to also take into account cases where the last line ended on the opening `(`, `[` or `{` and adds a regression test.

byteify-strings: add test for byteify-strings.py This tests the basic features expected from this script, some cases may not be covered yet. A future patch will demonstrate an issue with multi-line `(`, `[` and `{` alignment and propose a fix.

merge with stable

automation: push changes affecting .hgtags When I went to build the 5.1 tag using the in-repo automation, the automatic version calculation failed to deduce the clean 5.1 version string because we had only pushed the changeset corresponding to the 5.1 tag and not the changeset containing the 5.1 tag. So from the perspective of the remote repo, the 5.1 tag didn't exist yet and automatic version deduction failed. This commit changes the `hg push` to also push all changesets affecting the .hgtags file, ensuring the remote has up-to-date tags information. I tested this by creating a local draft changeset with a dummy tag value on a different DAG head and instructed the automation to build a revision that didn't have this change to .hgtags. The tag was successfully pushed and the built package had a version number incorporating that tag. Sending this to stable so the 5.1.1 automation hopefully "just works."

byteify-strings: add cli argument to handle `attr*()` when they are methods Certain code bases have useful utils that wrap the builtin functions, and are called like `util.setattr`.

bookmarks: actual fix for race condition deleting bookmark This is a simple but efficient fix to prevent the issue tested in `test-bookmarks-corner-case.t`. It might be worth pursuing a more generic approach where filecache learn to depend on each other, but that would not be suitable for stable. The issue is complicated enough that I documented the race and its current solution as inline comment. See this comment for details on the fix.

strip: access bookmark before getting a reference to changelog Bookmark access might invalidate the current changelog (to make sure both are in a reasonable synchronisation state). So we should grab the reference to changelog after we access bookmark. Otherwise we risk using a dead object for the whole strip process. (note: this dead object business probably requires a new layers of checking)

test: use a more verbose output in the test While debugging some test failure, I released the test never checks if the relevant changesets were preserved. So I am updating the test from `hg parents` usage to `hg log -G` with a special template. This increase the area covered by the test and clarify the test failures.

byteify-strings: simplify default value for `--treat-as-kwargs`

byteify-strings: add --treat-as-kwargs argument to handle kwargs-like objects This argument will help extensions move to Python 3 as keyword arguments should not be byte-prefixed. Most of the time, code bases will call this object `kwargs`, but other conventions exist like `opts`, so it should make sense to allow for custom names. This is a best effort solution that does minimal static checking; cases like `options = [o for o in ('a', 'b', 'c') if kwargs.get(o)]` and other just as complicated will not be detected.

byteify-strings: add helpers to check for item access or method call These helpers will be used in a future patch, split for ease of review.

byteify-strings: add support for ignore comments Our simple token analysis is sometimes not clever enough, we need to be able to turn off our script for parts of the code. This change introduces three special comments: - `#no-py3-transform` to tell `byteify-strings` ignore the next line - `#py3-transform: off` to ignore everything until the end of the file - `#py3-transform: on` to stop ignoring The last two can be particularly useful within Python 2/3 compatibility files.

byteify-strings: handle triple quoted strings if they are not docstrings As with anything in this script, this is a best effort approach. Most of the time, when a triple quoted string is assigned to something, it's not a docstring.

byteify-strings: handle multi-line strings in _ensuresysstr The current implementation did not handle calls like `repo.ui.log("first line" "other line")` correctly.

fix: run fixer tools in the repo root as cwd so they can use the working copy This lets fixer tools do things like find configuration files, with the caveat that they'll only see the version of that file in the working copy, regardless of what revisions are being fixed. Differential Revision: https://phab.mercurial-scm.org/D6440

config: add defaultvalue template keyword This patch tries to fix one of the issues mentioned in issue6014. This adds a new `defaultvalue` template keyword to be used with `hg showconfig` to get the default value of the config item. Differential Revision: https://phab.mercurial-scm.org/D6704

merge with stable

module-policy: update rust extension import to use the new module policy Differential Revision: https://phab.mercurial-scm.org/D6677

transaction: leave unfinished without crashing when not properly released I think the transaction.__del__ is there just as a last resort in case we (or an extension) forgot to release the transaction. When that happens, the repo can (or will on Python 3?) get deleted before the transaction. This leads to a crash in test-devel-warnings.t on Python 3 because we tried to access repo.dirstate, where repo was retried from a weak reference. There's not much we can do here, but let's at least avoid the crash. The user will have run `hg recover` afterwards regardless. Differential Revision: https://phab.mercurial-scm.org/D6664

unshelve: add abort on using continue and interactive together `unshelve --continue --interactive` will not work as expected by the user as the mode of in-progress unshelve is preserved and cannot be overwritten. This patch makes `unshelve` to throw an error on using both `--continue` and `--interactive` together with `unshelve`. Differential Revision: https://phab.mercurial-scm.org/D6703

py3: add one more test to list of passing tests Found by buildbot. Differential Revision: https://phab.mercurial-scm.org/D6702