tests: drop py2 support from test-doctest.py

tests: drop py2 support from test-demandimport.py

tests: optional PEP 657 error location in test-lfs-serve-access.t (issue6782)

tests: optional PEP 657 error location in test-extension.t (issue6781)

tests: filter out PEP 657 error locations in tracebacks (issue6780) https://docs.python.org/3/whatsnew/3.11.html#whatsnew311-pep657

pytype: add coverage for parts of hgext There are tons of things to fix here (which have been blacklisted for now), but this should help prevent further regressions.

convert: stop passing str to the dateutil API in darcs I'm sure there's a bunch more stuff in here that's broken, but this was flagged by pytype.

typing: suppress a couple of attribute-errors in convert I thought these might be real issues, but they're not.

hooklib: force an exception wrapped by errors.Abort to bytestr Flagged by PyCharm and pytype.

notify: force an exception wrapped by errors.Abort to bytestr Flagged by PyCharm and pytype.

typing: disable a bogus attribute-error warning in phabricator In a local pytype run, this fixes: File "/mnt/c/Users/Matt/hg/hgext/phabricator.py", line 359, in <lambda>: No attribute 'items' on bytes [attribute-error] In Union[Any, bytes] Called from (traceback): line 363, in process The `bytes` case takes the previous `if` branch though.

sparse: fix a py2 based usage of `map()` In a local pytype run, this fixes: File "/mnt/c/Users/Matt/hg/hgext/sparse.py", line 386, in debugsparse: unsupported operand type(s) for item retrieval: 'fcounts: Iterator[int]' and '0: int' [unsupported-operands] No attribute '__getitem__' on 'fcounts: Iterator[int]' File "/mnt/c/Users/Matt/hg/hgext/sparse.py", line 387, in debugsparse: unsupported operand type(s) for item retrieval: 'fcounts: Iterator[int]' and '1: int' [unsupported-operands] No attribute '__getitem__' on 'fcounts: Iterator[int]' File "/mnt/c/Users/Matt/hg/hgext/sparse.py", line 388, in debugsparse: unsupported operand type(s) for item retrieval: 'fcounts: Iterator[int]' and '2: int' [unsupported-operands] No attribute '__getitem__' on 'fcounts: Iterator[int]'

typing: adjust `mercurial.util.iterlines()` to accept any `Iterable` In a local pytype run on the extensions, this fixes: File "/mnt/c/Users/Matt/hg/hgext/phabricator.py", line 788, in maketext: Function mercurial.util.iterlines was called with the wrong arguments [wrong-arg-types] Expected: (iterator: Iterator[bytes]) Actually passed: (iterator: list) Attributes of protocol Iterator[bytes] are not implemented on list: __next__

typing: disable an attribute-error warning in the journal extension The code is complicated enough that pytype doesn't realize that `name` can't be `None` if it is evaluated here.

remotefilelog: byteify the message for a few StorageErrors Flagged by pytype locally.

histedit: byteify the help for the multifold action While there's some allowance for str in `_()`, it's commented to be for "goofy unicode docstrings in test", so no idea how well that works, but it should at least come back as bytes. With HGPLAIN, however, the str isn't touched and is returned as-is, so this seems like a real bug.

typing: disable a few incorrect warnings in pywatchman The module-attr warnings are for things that only exist on Windows, and the wrong-keyword-args warning is due to a special case for a specific constructor. Both of these are properly conditionalized.

watchman: refactor transport connecting to unconfuse pytype Pytype sees `self.proc` as potentially `None` here, even though it's set by the `_connect()` logic. Instead of asserting, simply use the process returned by that method (which it sets into `self.proc` before returning, so there's no functional change here).

watchman: refactor `ctypes.windll.kernel32` references to a local variable This is flagged by pytype as an attribute-error, and it's easier to disable that in a single place.

typing: disable [unsupported-operands] warning in the largefiles outgoing hook For some reason, pytype thinks `toupload` is a set: No attribute '__setitem__' on Set[nothing] (It actually is a set in the subsequent `else` branch, but I'm not interested in trying to rewrite this to be consistent.)

typing: add some assertions that a variable isn't None In the case of blackbox, there's a default limit if one isn't explicitly supplied. For the monotone regex, neither group is optional, so a match means it's not None.

largefiles: reference `mercurial.configitems.dynamicdefault` directly Pytype was unable to see `dynamicdefault` on `eh.configitem`. This is clearer anyway.

releasenotes: fix a typo in a comment

schemes: fix a broken check for drive letter conflicts Flagged by pytype locally. It appears to have regressed in 1863584f2fba (not yet released). This seems like an obvious typo- `dict.isalpha()` is nonsense. There's no crash though because `schemes` is pre-populated with 5 schemes (that are all now defunct), so the length of the dict is never 1, so it's impossible to abort.

typing: suppress a bunch of potential import-error cases in extensions As flagged by pytype locally. Either the ImportError is locally handled, or the imported module was previously determined to be present by `hgave` (for the phabricator extension), or is handled by the `hgext.convert.subversion` module when imported (for the `hgext.convert.transport` module).

watchman: drop some py2 compat code The `unicode` reference was being flagged by pytype, even though it was never evaluated on py3. There's more that can be dropped and `compat.py` can probably be inlined if we don't care about minimizing the code changes from FB. But I don't feel like dealing with that.

pytype: add coverage for hgdemandimport This would have flagged what needed fixing in 48e38b179106 long ago.

hgweb: skip body creation of HEAD for most requests The body is thrown away anyway, so this just wastes a lot of CPU time. In the case of /archive/, this skips manifest processing and the actual file archiving, resulting in a huge difference. The most tricky part here is skipping the Content-Length creation as it would indicate the output size for the corresponding GET request.

branching: merge stable into default

urlutil: drop the deprecated `getpath()` This was deprecated in 5.9.

ui: drop the deprecated `getpath()` This was deprecated in 5.9.

ui: drop the deprecated `expandpath()` This was deprecated since 5.8.

relnotes: last-minute addition to 6.3.2

relnotes: add 6.3.2

Added signature for changeset 59466b13a3ae

Added tag 6.3.2 for changeset 59466b13a3ae

filemerge: fix crash when using filesets in [partial-merge-tools] Without this patch, you'd get `mercurial.error.ProgrammingError: fileset expression with no context`.

packaging: add dependencies to the PyOxidizer build on macOS Otherwise, we get a bunch of test failures for missing things like pygments, or tests skipped entirely. The input file is a copy/paste from the equivalent Windows file, but with dulwich, pygit2, and pytest-vcr commented out because the build process errors out with them, flagging them as incompatible with loading from memory. I have no idea if that's actually true or not, because I've noticed that if I don't `make clean` after every build, the next build flags the watchman stuff as incompatible with loading from memory. The remaining failures are: Failed test-alias.t: output changed Failed test-basic.t: output changed Failed test-check-help.t: output changed Failed test-commit-interactive.t: output changed Failed test-extension.t: output changed Failed test-help.t: output changed Failed test-i18n.t: output changed Failed test-log.t: output changed Failed test-qrecord.t: output changed Failed test-share-safe.t: output changed Most of the issues seem related to loading help for disabled extensions from `hgext.__index__`, namely the full extension help being unavailable, not being able to resolve what commands are provided by what extension, and not having the command level help available. test-log.t, test-commit-interactive.t, and test-i18n.t look like i18n (or lack thereof) issues. test-basic.t is just odd: @@ -55,7 +55,7 @@ On Python 3, stdio may be None: $ hg debuguiprompt --config ui.interactive=true 0<&- - abort: Bad file descriptor (no-rhg !) + abort: response expected abort: response expected (rhg !) [255] $ hg version -q 0<&-

tests: conditionalize path output for in-memory pyoxidizer resources

tests: conditionalize test output for in-filesystem pyoxidizer resources The in-memory pyoxidizer builds apparently behave as expected.

hghave: add predicates for embedded and filesystem pyoxidizer resources There are a handful of tests with different output between the two flavors of pyoxidizer builds (like the location of the modules and templates), and a few others that avoid `known-bad-output` cases with the embedded resources that shouldn't cause the tests to fail.

run-tests: support --pyoxidized on macOS We should definitely be able to test this before releasing it with an installer.

make: add a target for building pyoxidizer tests on macOS The resources seem to be embedded inside the binary, but for some reasons they aren't read there. And since they are embedded, they aren't staged by the build in the `lib` directory like on Windows. So copy them from the repo. We can figure out what's going wrong later.

copyright: update to 2023

bisect: correct message about aborting an in-progress bisect (issue6527) By using a custom cmdhint message here we're avoiding automatic hint generation in _statecheck.hint(), which would suggest 'hg bisect --continue' and 'hg bisect --abort' (neither of which is a valid option). This patch is only fixing the message about in-progress bisect, it doesn't modify the unfinished state checking logic.

revlog: fix misleading comment about _maxinline

verify: print short `p1` node in relevant dirstate messages This will help with debugging.

verify: also check dirstate The dirstate already is capable of verifying its integrity (although v2 features are not yet checked), let's run that code in `hg verify`.

tests: use the `--quiet` flag for verify when applicable This reduces a lot of the test output that was otherwise useless, and also makes it a lot easier to add things to verify without breaking the test suite because of additional output.

verify: format messages directly at the source

dirstate: update messages in verify to not use the old `state` API

dirstate: add narrow support to `verify` This will be called later in the series by the `verify` command.

dirstate: stop using `entry.state()` for logic in `verify` Use the new API instead of the deprecated one. The next changeset will remove the use of `state` in this method altogether (currently still in the messages)

dirstate-entry: add `modified` property This was already done in the Rust implementation and is a useful primitive. The C implementation had this called `merged`, but wasn't used anywhere. It will be used in the next changeset.

debug: add debug-revlog-stats command Display statistics about revlogs in the store. Useful to get an approximate size of a repository, etc. More statistics will be added in the future.

typing: attempt to remove @overloads in the platform module for stdlib methods This is mostly successful, as examining util.pyi, posix.pyi, and windows.pyi after a pytype run shows that the type overloads for `oslink`, `readlink`, `removedirs`, `rename`, `split`, and `unlink` have been removed. (Some of these still have an @overload, but the differences are the variable names, not the types.) However, @overloads remain for `abspath` and `normpath` for some reason. It's useful to redefine these methods for the type checking phase because in addition to excluding str and PathLike variants, some of these functions have optional args in stdlib that aren't implemented in the custom implementation on Windows, and we want the type checking to flag that instead of assuming it's an allowable overload everywhere. One last quirk I noticed that I can't explain- `pycompat.TYPE_CHECKING` is always False, so the conditionals need to check `typing.TYPE_CHECKING` directly. I tried dropping the custom code for assigning `pycompat.TYPE_CHECKING` and simply did `from typing import TYPE_CHECKING` directly in pycompat.py, and used `pycompat.TYPE_CHECKING` for the conditional here... and pytype complained that `pycompat` doesn't have the `TYPE_CHECKING` variable.

typing: add trivial type hints to rest of the windows platform module Skipping the file wrappers for now because there's interplay with C code, and making them subclass `typing.BinaryIO_Proxy` confuses PyCharm a bit.

typing: add type hints to the rest of the posix module These methods either don't have an analog in the windows module, or are aliased in the windows module from something else (like os.path.xxx).

typing: add type hints to the platform `cachestat` classes

util: fix the signature of observedbufferedinputpipe._fillbuffer() Flagged by PyCharm, since it didn't match the signature of the method being overridden. The default value in the superclass is also `_chunksize`, and I suspect that the amount read from `osread` should be limited to what is passed in. Only one caller (`bufferedinputpipe.unbufferedread()`) passes this argument, and it passes the max of `_chunksize` and whatever it was passed.

tests: drop some obsolete py2 handling in util.py doctest Flagged by PyCharm while inspecting imports from the platform modules.

typing: add type hints to the common posix/windows platform functions These are done in sync because some platforms have empty implementations, and it isn't obvious what the types should be without examining the other. We want the types aligned, so @overload definitions that differ aren't generated. The only differences here are the few methods that unconditionally raise an error are marked as `NoReturn`, which doesn't seem to bother pytype. A couple of the posix module functions needed to be updated with a modern ternary operator, because pytype seems to want to use the type of the second object in the old `return x and y` style.

typing: add type hints to the posix platform module matching win32.py

typing: add type hints to mercurial/win32.py These are the low level functions that are imported by the mercurial.windows module, which is in turn imported by mercurial.utils as the platform module. Pretty straightforward, but pytype inferred very little of it, likely because of the heavy ctypes usage. It also seems to trigger a pytype bug in procutil, now that it has an idea of the underlying function type, so disable that warning to maintain a working test.

windows: drop some py2 registry module importing The comment was actually backwards- `winreg` is importable on py3, and is already imported by mercurial/windows.py.

typing: add type hints to the platform specific scm modules Surprisingly, pytype struggled to figure out the return types in the posix functions.

typing: add type hints to most mercurial/pycompat.py functions The `rapply` methods are left out because it's not `rapply(f, xs: _T0) -> _T0` as I first thought- it's used somewhere to walk a collection and convert between bytes and str. Also, the `open()` call is partially untyped because I'm not sure what its purpose is at this point- both the name and mode can be either bytes or str as it is currently constituted. It might make sense to assert that the file is being opened in binary mode (like `namedtempfile()`) and cast the result to `BinaryIO`, but that shouldn't be smuggled in with these other changes. The return is currently typed as `Any` because something suddenly got smarter and a few uses in util.py (like readfile()) suddenly think it returns `IO[str]` instead of `IO[bytes]` (BinaryIO), and it flags the type mismatch there.

statprof: don't pass str `sys.argv` to a function expecting bytes Found by typing the global functions in mercurial.pycompat.

typing: drop an unnecessary warning disabling comment in match.py This stopped being necessary in d2e1dcd4490d, when the exception stopped being subscripted.

scmposix: don't subscript IOError This warning disabling has been in place since late 2019 in 667f56d73ceb. We should have had some py3 support at the time, but both pytype complains and subscripting a real FileNotFoundError generated in `hg debugshell` crashed, so maybe this fixes a problem. It looks like all other instances of subscripting exceptions have been replaced (at least as far as greping for `== errno.` revealed).

typing: add type hints to pycompat.bytestr The problem with leaving pytype to its own devices here was that for functions that returned a bytestr, pytype inferred `Union[bytes, int]`. It now accepts that it can be treated as plain bytes. I wasn't able to figure out the arg type for `__getitem__`- `SupportsIndex` (which PyCharm indicated is how the superclass function is typed) got flagged: File "/mnt/c/Users/Matt/hg/mercurial/pycompat.py", line 236, in __getitem__: unsupported operand type(s) for item retrieval: bytestr and SupportsIndex [unsupported-operands] Function __getitem__ on bytestr expects int But some caller got flagged when I marked it as `int`. There's some minor spillover problems elsewhere- pytype doesn't seem to recognize that `bytes.startswith()` can optionally take a 3rd and 4th arg, so those few places have the warning disabled. It also flags where the tar API is being abused, but that would be a tricky refactor (and would require typing extensions until py3.7 is dropped), so disable those too.

pycompat: explicitly prefix builtin attr usage with `builtins.` It doesn't seem like this would fix any bug, because the wrapped functions that take bytes instead of str are defined after these calls. But PyCharm was flagging the second and third uses, saying "Type 'str' doesn't have expected attribute 'decode'". It wasn't flagging the first, but I changed it for consistency.

typing: add type hints to global variables in mercurial/pycompat.py The way `osaltsep` and `sysexecutable` were defined, pytype determined them to be `Union[bytes, str]`. This was a problem because that cascaded to all of the callers, and also because it couldn't be annotated as bytes on the initial assignment. Therefore, we use a ternary operator. The documentation says that `sys.executable` can either be None or an empty string if the value couldn't be determined. We opt for an empty string here because there are places that blindly pass it to `os.path.xxx()` functions, which crash if given None. Other places test `if pycompat.sysexecutable`, so empty string works for both.

windows: drop an unused method The only caller was removed in 563eb25e079b.

typing: add type hints to the prompt methods in mercurial/ui.py The @overloads allow for the callers that pass a non-None `default` to not have to worry about handling a None return to appease pytype.

ui: split the `default` arg out of **kwargs for the internal prompt method This arg was required anyway, based on how it was accessed. Having it separate allows it to be typed though, and this will simplify things for the callers- if a non-None `default` is passed, the return can never be None. That can be expressed with `@overload` when the arg can be typed, but that's not possible when it is rolled up in **kwargs. The default value is simply copied from the public `prompt()` above it.

typing: add trivial type hints to mercurial/ui.py There's not really a pattern here; it's mostly obvious return types and in a few cases, obvious parameter types. Some other "obvious" functions are left out because of quirks in how the return value for the various config() functions are inferred cause pytype to complain.

doc: don't pass str to ui methods in check-seclevel.py

typing: add type hints related to message output in mercurial/ui.py This will shake loose some bytes vs str issues in the doc checker.

typing: add type hints related to progress bars in mercurial/ui.py Pretty low hanging fruit while trying to deal with other more complicated parts of this module.

pytype: stop excluding mercurial/ui.py ui.extractchoices() is perhaps making assumptions that it shouldn't about the pattern always matching, but presumably we have test coverage for that. PyCharm flags the updated classes with a warning "Class xxx must implement all abstract methods", and suggests adding `abc.ABC` to the superclasses. I'm not sure why, unless it doesn't recognize the `__getattr__()` delegation pattern. Additionally, we can't unconditionally subclass `typing.BinaryIO` because that defeats the `__getattr__` delegation to the wrapped object at runtime. Instead, it has to only subclass during the type checking phase[1]. In any event, this fixes: File "/mnt/c/Users/Matt/hg/mercurial/ui.py", line 1518, in _runpager: Function subprocess.Popen.__new__ was called with the wrong arguments [wrong-arg-types] Expected: (cls, args, bufsize, executable, stdin, stdout: Optional[Union[IO, int]] = ..., ...) Actually passed: (cls, args, bufsize, stdin, stdout: Union[mercurial.utils.procutil.WriteAllWrapper, mercurial.windows.winstdout], ...) File "/mnt/c/Users/Matt/hg/mercurial/ui.py", line 1798, in extractchoices: No attribute 'group' on None [attribute-error] In Optional[Match[bytes]] File "/mnt/c/Users/Matt/hg/mercurial/ui.py", line 1799, in extractchoices: No attribute 'group' on None [attribute-error] In Optional[Match[bytes]] [1] https://stackoverflow.com/q/71365594

tests: document what the other currently enabled pylint check ensures

hghave: detect newer pylint Older versions (e.g. 2.7.2) say: "Usage: pylint [options]" Newer versions (e.g. 2.15.5) say: "usage: pylint [options]"

emitrevision: consider ancestors revision to emit as available base This should make more delta base valid. This notably affects: * case where we skipped some parent with empty delta to directly delta against an ancestors * case where an intermediate snapshots is stored. This change means we could sent largish intermediate snapshots over the wire. However this is actually a sub goal here. Sending snapshots over the wire means the client have a high odd of simply storing the pre-computed delta instead of doing a lengthy process that will… end up doing the same intermediate snapshot. In addition the overall size of snapshot (or any level) is "only" some or the overall delta size. (0.17% for my mercurial clone, 20% for my clone of Mozilla try). So Sending them other the wire is unlikely to change large impact on the bandwidth used. If we decide that minimising the bandwidth is an explicit goal, we should introduce new logic to filter-out snapshot as delta. The current code has no notion explicite of snapshot so far, they just tended to fall into the wobbly filtering options. In some cases, this patch can yield large improvement to the bundling time: ### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog # benchmark.name = perf-bundle # benchmark.variants.revs = last-100000 before: 68.787066 seconds after: 47.552677 seconds (-30.87%) That translate to large improvement to the pull time : ### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog # benchmark.name = pull # benchmark.variants.issue6528 = disabled # benchmark.variants.revs = last-100000 before: 142.186625 seconds after: 75.897745 seconds (-46.62%) No significant negative impact have been observed.

sqlitestore: add an `ancestors` method We will need it during bundling. The implementation mirror the one in revlog.

emitrevision: if we need to compute a delta on the fly, try p1 or p2 first Falling back to `prev` does not yield any real value on modern storage and result in pathological changes to be created on the other side. Doing a delta against a parent will likely be smaller (helping the network) and will be safer to apply on the client (helping future pulls by Triggering intermediate snapshop where they will be needed by later deltas).

emitrevision: simplify the fallback to computed delta Not using the stored delta, or having a full snapshot on disk behave the same ways, so lets use the same code path for that, this is simpler, and it update will be simpler.

emitrevision: also check the parents in the availability closure One of the point of having a closure is to gather the logic in it. So we gather the logic. The `parents[:]` part is a bit ugly but will be replaced by better code soon anyway.

emitrevision: add a small closure to check if a base is usable We will make more use of this and make it more complex too.

chg: scale the timeout in test with the rest This should avoid some flakiness where the logs reports server shutting down.

bundle: emit full snapshot as is, without doing a redelta With the new `forced` delta-reused policy, it become important to be able to send full snapshot where full snapshot are needed. Otherwise, the fallback delta will simply be used on the client side… creating monstrous delta chain, since revision that are meant as a reset of delta-chain chain becoming too complex are simply adding a new full delta-tree on the leaf of another one. In the `non-forced` cases, client process full snapshot from the bundle differently from deltas, so client will still try to convert the full snapshot into a delta if possible. So this will no lead to pathological storage explosion. I have considered making this configurable, but the impact seems limited enough that it does not seems to be worth it. Especially with the current sparse-revlog format that use "delta-tree" with multiple level snapshots, full snapshot are much less frequent and not that different from other intermediate snapshot that we are already sending over the wire anyway. CPU wise, this will help the bundling side a little as it will not need to reconstruct revisions and compute deltas. The unbundling side might save a tiny amount of CPU as it won't need to reconstruct the delta-base to reconstruct the revision full text. This only slightly visible in some of the benchmarks. And have no real impact on most of them. ### data-env-vars.name = pypy-2018-08-01-zstd-sparse-revlog # benchmark.name = perf-bundle # benchmark.variants.revs = last-40000 before: 11.467186 seconds just-emit-full: 11.190576 seconds (-2.41%) with-pull-force: 11.041091 seconds (-3.72%) # benchmark.name = perf-unbundle # benchmark.variants.revs = last-40000 before: 16.744862 just-emit-full:: 16.561036 seconds (-1.10%) with-pull-force: 16.389344 seconds (-2.12%) # benchmark.name = pull # benchmark.variants.revs = last-40000 before: 26.870569 just-emit-full: 26.391188 seconds (-1.78%) with-pull-force: 25.633184 seconds (-4.60%) Space wise (so network-wise) the impact is fairly small. When taking compression into account. Below are tests the size of `hg bundle --all` for a handful of benchmark repositories (with bzip, zstd compression and without it) This show a small increase in the bundle size, but nothing really significant except maybe for mozilla-try (+12%) that nobody really pulls large chunk of anyway. Mozilla-try is also the repository that benefit the most for not having to recompute deltas client size. ### mercurial: bzip-before: 26 406 342 bytes bzip-after: 26 691 543 bytes +1.08% zstd-before: 27 918 645 bytes zstd-after: 28 075 896 bytes +0.56% none-before: 98 675 601 bytes none-after: 100 411 237 bytes +1.76% ### pypy bzip-before: 201 295 752 bytes bzip-after: 209 780 282 bytes +4.21% zstd-before: 202 974 795 bytes zstd-after: 205 165 780 bytes +1.08% none-before: 871 070 261 bytes none-after: 993 595 057 bytes +14.07% ### netbeans bzip-before: 601 314 330 bytes bzip-after: 614 246 241 bytes +2.15% zstd-before: 604 745 136 bytes zstd-after: 615 497 705 bytes +1.78% none-before: 3 338 238 571 bytes none-after: 3 439 422 535 bytes +3.03% ### mozilla-central bzip-before: 1 493 006 921 bytes bzip-after: 1 549 650 570 bytes +3.79% zstd-before: 1 481 910 102 bytes zstd-after: 1 513 052 415 bytes +2.10% none-before: 6 535 929 910 bytes none-after: 7 010 191 342 bytes +7.26% ### mozilla-try bzip-before: 6 583 425 999 bytes bzip-after: 7 423 536 928 bytes +12.76% zstd-before: 6 021 009 212 bytes zstd-after: 6 674 922 420 bytes +10.86% none-before: 22 954 739 558 bytes none-after: 26 013 854 771 bytes +13.32%

bundle: when forcing acceptance of incoming delta also accept snapshot Snapshot where never considered reusable and the unbundling side always tried to find a delta from them. In the `forced` mode this is counter-productive because it will either connect two delta-tree that should not be connected or it will spend potentially a lot of time because creating a full snapshot anyway. So in this mode, we accept the full snapshot as is. This changeset is benchmarked with its children so please do not split them apart when landing.

delta-find: properly report full snapshot used from cache as such The number of tries and the delta base is reported differently so we missed there detection initially.

test-acl: glob the payload size again This size of bundle-2 payload are irrelevant for this test and only appears in its output because other pieces of the debug output are important. We glob it these number before they get in our way again.

hghave: we might need py310 and py311 at some point Some tests are already showing slightly different results on Python 3.11. The better idea would be to make them more portable, but if that's not possible, now we can use hghave detection for certain lines. I wonder if there will ever be Python 31.0 and 31.1 though.

hghave: detect Python 3.10 and 3.11 as well Noticed because test-contrib-relnotes.t was skipped.

extensions: load help from hgext.__index__ as a fallback this time Prior to 843418dc0b1b, `hgext.__index__` was consulted first if present, which caused the longer help from the extension modules to be ignored, even when available. But that change causes a bunch of test failures when the pyoxidized binary bundles *.pyc in the binary, saying the there's no help topic for `hg help $disabled_extension` and suggesting the use of `--keyword`, rather than showing a summary and indicating that it is disabled. Current failures were in test-check-help.t, test-extension.t, test-help.t, and test-qrecord.t. Ideally, we would read the various *.pyc files from memory and slurp in the docstring, but I know that they used to not be readable as resources, and I can't figure out how to make it work now. So maybe 3.9 and/or the current PyOxidizer doesn't support it yet. I got closer in py2exe with `importlib.resources.open_binary("hgext", "rebase.pyc")`, but `open_binary()` on *.pyc fails in pyoxidizer.[1] Either way, the *.pyc can't be passed to `ast.parse()` as `extensions._disabledcmdtable()` is doing, so I'm setting that aside for now. [1] https://github.com/indygreg/PyOxidizer/issues/649

extensions: process disabled external paths when `hgext` package is in-memory This fixes `hg help -e ambiguous` in test-helpt.t:2055 with the `ambiguous = !./ambiguous.py` configuration, when `hgext` is not in the filesystem (e.g. pyoxidizer builds with in-memory resources, or TortoiseHg with py2exe), but the disabled external extension is. Now instead of aborting with a suggestion to try `--keyword`, the help command prints text for the extension.

hg: show the correct message when cloning an LFS repo with extension disabled The `extensions._disabledpaths()` doesn't handle fetching help from `__index__`, so it returns an empty dictionary of paths. That means None is always returned from `extensions.disabled_help()` when embedding resources inside the pyoxidizer or py2exe binary, regardless of the arg or if is an external extension stored in the filesystem. And that means wrongly telling the user with an explicitly disabled LFS extension that it will be enabled locally upon cloning from an LFS remote. That causes test-lfs-serve.t:295 to fail. This effectively reverts most of the rest of 843418dc0b1b, while keeping the help text change in place (which was specifically identified as a problem).

demandimport: fix a crash in LazyFinder.__delattr__ I was tinkering with `with hgdemandimport.deactivated()` wrapped around loading the keyring module, and got spew that seemed to be confirmed by PyCharm. But I can't believe we haven't seen this before (and phabricator uses the same pattern): ** Unknown exception encountered with possibly-broken third-party extension "mercurial_keyring" 1.4.3 (keyring 23.11.0, backend unknown) ** which supports versions unknown of Mercurial. ** Please disable "mercurial_keyring" and try your action again. ** If that fixes the bug please report it to https://foss.heptapod.net/mercurial/mercurial_keyring/issues ** Python 3.9.15 (main, Oct 13 2022, 04:28:25) [GCC 7.5.0] ** Mercurial Distributed SCM (version 6.3.1) ** Extensions loaded: absorb, attorc 20220315, blackbox, eol, extdiff, fastannotate, lfs, mercurial_keyring 1.4.3 (keyring 23.11.0, backend unknown), phabblocker 20220315, phabricator 20220315, purge, rebase, schemes, share, show, strip, uncommit Traceback (most recent call last): File "/usr/local/bin/hg", line 59, in <module> dispatch.run() File "/usr/local/lib/python3.9/site-packages/mercurial/dispatch.py", line 143, in run status = dispatch(req) File "/usr/local/lib/python3.9/site-packages/mercurial/dispatch.py", line 232, in dispatch status = _rundispatch(req) File "/usr/local/lib/python3.9/site-packages/mercurial/dispatch.py", line 276, in _rundispatch ret = _runcatch(req) or 0 File "/usr/local/lib/python3.9/site-packages/mercurial/dispatch.py", line 451, in _runcatch return _callcatch(ui, _runcatchfunc) File "/usr/local/lib/python3.9/site-packages/mercurial/dispatch.py", line 461, in _callcatch return scmutil.callcatch(ui, func) File "/usr/local/lib/python3.9/site-packages/mercurial/scmutil.py", line 153, in callcatch return func() File "/usr/local/lib/python3.9/site-packages/mercurial/dispatch.py", line 441, in _runcatchfunc return _dispatch(req) File "/usr/local/lib/python3.9/site-packages/mercurial/dispatch.py", line 1265, in _dispatch return runcommand( File "/usr/local/lib/python3.9/site-packages/mercurial/dispatch.py", line 899, in runcommand ret = _runcommand(ui, options, cmd, d) File "/usr/local/lib/python3.9/site-packages/mercurial/dispatch.py", line 1277, in _runcommand return cmdfunc() File "/usr/local/lib/python3.9/site-packages/mercurial/dispatch.py", line 1263, in <lambda> d = lambda: util.checksignature(func)(ui, *args, **strcmdopt) File "/usr/local/lib/python3.9/site-packages/mercurial/util.py", line 1880, in check return func(*args, **kwargs) File "/root/mercurial_keyring/mercurial_keyring/mercurial_keyring.py", line 962, in cmd_keyring_check user, pwd, source, final_url = handler.get_credentials( File "/root/mercurial_keyring/mercurial_keyring/mercurial_keyring.py", line 497, in get_credentials keyring_pwd = password_store.get_http_password(keyring_url, actual_user) File "/root/mercurial_keyring/mercurial_keyring/mercurial_keyring.py", line 287, in get_http_password return self._read_password_from_keyring( File "/root/mercurial_keyring/mercurial_keyring/mercurial_keyring.py", line 335, in _read_password_from_keyring keyring = import_keyring() >> `with hgdemandimport.deactivated()` inserted here File "/root/mercurial_keyring/mercurial_keyring/mercurial_keyring.py", line 120, in import_keyring return _import_keyring() File "/root/mercurial_keyring/mercurial_keyring/mercurial_keyring.py", line 133, in _import_keyring mod, was_imported_now = meu.direct_import_ext( File "/usr/lib/python3.9/site-packages/mercurial_extension_utils.py", line 1381, in direct_import_ext __import__(module_name) File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "/usr/local/lib/python3.9/site-packages/hgdemandimport/demandimportpy3.py", line 46, in exec_module self.loader.exec_module(module) File "/usr/lib/python3.9/site-packages/keyring/__init__.py", line 1, in <module> from .core import ( File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "/usr/local/lib/python3.9/site-packages/hgdemandimport/demandimportpy3.py", line 46, in exec_module self.loader.exec_module(module) File "/usr/lib/python3.9/site-packages/keyring/core.py", line 11, in <module> from . import backend, credentials File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "/usr/local/lib/python3.9/site-packages/hgdemandimport/demandimportpy3.py", line 46, in exec_module self.loader.exec_module(module) File "/usr/lib/python3.9/site-packages/keyring/backend.py", line 13, in <module> from .py312compat import metadata File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "/usr/local/lib/python3.9/site-packages/hgdemandimport/demandimportpy3.py", line 46, in exec_module self.loader.exec_module(module) File "/usr/lib/python3.9/site-packages/keyring/py312compat.py", line 10, in <module> import importlib_metadata as metadata # type: ignore File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "/usr/local/lib/python3.9/site-packages/hgdemandimport/demandimportpy3.py", line 46, in exec_module self.loader.exec_module(module) File "/usr/lib/python3.9/site-packages/importlib_metadata/__init__.py", line 715, in <module> class MetadataPathFinder(NullFinder, DistributionFinder): File "/usr/lib/python3.9/site-packages/importlib_metadata/_compat.py", line 24, in install disable_stdlib_finder() File "/usr/lib/python3.9/site-packages/importlib_metadata/_compat.py", line 43, in disable_stdlib_finder del finder.find_distributions File "/usr/local/lib/python3.9/site-packages/hgdemandimport/demandimportpy3.py", line 88, in __delattr__ return delattr(object.__getattribute__(self, "_finder")) TypeError: delattr expected 2 arguments, got 1

tests: use `test -f` instead of `ls` to see if a file is present (issue6662) ls's exit code when file doesn't exist is 2 on Linux and 1 on NetBSD, so let's use something that's supposedly more portable, since we only care whether the file is there or not.

rust-status: fix thread count ceiling This was forcing 16 threads instead of creating a ceiling, which is wrong when either the available parallelism of the platform is lower or when the user wants to set it explicitly (like we do in `run-tests.py`)

amend: add a --draft option to set phase to draft Some users create commits in secret phase by default and then want to make them draft so they can be uploaded. This patch adds a --draft option for that. We already have a flag for changing the phase to secret, so it seems consistent to have one for draft.

commit: add --draft option to use draft phase

tests: use graph log in test-phases.t It's hard to tell that the phases are ordered correctly without seeing the graph.

commit: move check for incompatible args earlier I think it makes sense to check the command line arguments as early as possible, so we don't have to wait for a repo lock to tell the user that they passed invalid arguments.

delta-find: add a delta-reuse policy that blindly accepts incoming deltas When this policy is set, incoming deltas are blindly accepted without regard for the validity of the chain they build.

delta-find: add a `delta-reuse-policy` on configuration `path` That option allows to control the behavior on a per-path basis, opening the way to treating pulls from central servers differently than other operations.

changegroup: add `delta_base_reuse_policy` argument The argument available through function from changegroup.apply to `revlog.apply` allow to override the revlog configuration in terms of delta-base-reuse policy when searching for a delta to store a revision. It will be put to use in the next changesets.

bundleoperation: optionnaly record the `remote` that produced the bundle We have the information at hand, and the peer now have knownledge of its `path` object, which constaints useful behavior configuration. So the simpler seems to be to pass that object around so it can be used if needed.

delta-find: add a test checking various simple behavior There are enough work happening in this area that it is worth having a dedicated test for it. So far we add to small test checking that the "best" parent is picked as the delta base and that this behavior can be controlled during commit and unbundle.

peer: pass the `path` to the statichttp peer We now have all peer able to receive and store a `path` object. Hooray.

peer: get the `path` object down to the sshpeer Same logic as the other peers.

logexchange: use the proper accessors to get the remote url There is an official method, let us use it. this will prevent a crash when the private attribute disappear.

peer: get the `path` object down to the httppeer One more peer with a path stored.

path: fix `url.copy` dropping the port The copy method have been wrong for a while, but the new code reveals it.

peer: pass the `path` object to `make_peer` We don't do anything with it yet, but we can start implementing it for each peer type starting now.

path: allow to copy a path while adjusting the url This will be used by `scheme` in the next changesets.

peer: store the path object used to build a peer from a repo This is the simplest case, we so starts with it.

peer: build a `path` object on the fly when needed So now, we always have a `path` object around when building the peer

peer: have `repo.peer` take an optional `path` argument We are ready to start to actually set the value now.

peer: add a `path` attribute to peer It will start being set in the coming changesets.

peer: have a common constructor and use it For now it does not do much, but we will extend it to also store a path object soon.

peer: use a dedicated name for the `peer` constructor We want to change the argument it takes, so we rather make them different function.

peer: dissolve `_peerlookup` into its last two callers This is about to need more changes and the function won't be useful. We do it early to clarify later changes.

peer: stop having a `peer()` method on `peer()` This is already a peer, why do you want a peer if you already have one.

clone: explicitly detect the need to fetch a peer Instead of having `peer()` method on all `peer()` for this usecase, we could simply handle it explicitly.

addbranchrevs: explicitly detect the need to fetch a peer Instead of having `peer()` method on all `peer()` for this usecase, we could simply handle it explicitly.

path: pass `path` to `peer` in `hg clone` We directly use the `path` object to build the `peer` object.

path: use `get_clone_path_obj` in share The return is simpler to use, and this mean less user for the old function. We directly use the `path` object to build the `peer` object.

path: pass `path` to `peer` in mq We directly use the `path` object to build the `peer` object. There is one case where we don't. We should fix that at the same time as we fix the sub-repo cases.

path: use `get_clone_path_obj` in _getlocal We don't need to feed the `path` object to a `peer` object, but using an higher level function is simpler here (e.g. no subscript access, explicit attribute access).

path: pass `path` to `peer` in `hg init` We directly use the `path` object to build the `peer` object.

path: add a `get_clone_path_obj` function Same logic as the `get_unique_pull_path_obj` function, this give access to the `path` object directly.

path: simplify the implementation of `get_clone_path` We can simply use the logic from `get_unique_pull_path_obj` now.

path: clarify document of `get_clone_path` This return a url as `bytes`, not a `path` object.

path: pass `path` to `peer` in `hg perf::discovery` We directly use the `path` object to build the `peer` object.

path: pass `path` to `peer` in remotefilelog's tests We directly use the `path` object to build the `peer` object.

path: pass `path` to `peer` in `hg fastannotate` We directly use the `path` object to build the `peer` object.

path: pass `path` to `peer` in infinitepush We directly use the `path` object to build the `peer` object.

path: pass `path` to `peer` in largefiles We directly use the `path` object to build the `peer` object.

path: pass `path` to `peer` in narrow We directly use the `path` object to build the `peer` object.

path: pass `path` to `peer` in `hg fetch` We directly use the `path` object to build the `peer` object.

path: use `get_unique_pull_path_obj` in `hg relink` This is not really needed, but that help removing caller of the older function.

path: pass `path` to `peer` in `hg transplant` We directly use the `path` object to build the `peer` object.

path: pass `path` to `peer` in `hg debugbackupbundle` We directly use the `path` object to build the `peer` object.

path: pass `path` to `peer` in `hg debugssl` We directly use the `path` object to build the `peer` object.

path: pass `path` to `peer` in `hg debugdiscovery` We directly use the `path` object to build the `peer` object.

path: pass `path` to `peer` in `remote(...)` revset We directly use the `path` object to build the `peer` object.

path: pass `path` to `peer` in `hg summary` We directly use the `path` object to build the `peer` object.

path: pass `path` to `peer` in `hg identify` We directly use the `path` object to build the `peer` object.

path: introduce a `get_unique_pull_path_obj` function Unlike the previous one, `get_unique_pull_path`, this function return the `path` object, opening more option for the caller. note that this highlight we don't actually need the `repo` argument to `get_pull_paths`, however changing the API would be annoying for third party extensions.

path: simplify the `get_unique_pull_path` function Simply delegate the search to `get_pull_paths` and check how many we got.

path: remove outdated documentation point from `get_unique_push_path` This is no longer true.

path: pass `path` to `peer` in `hg pull` We directly use the `path` object to build the `peer` object. We don't use it for sub-repositories yet, which will have to be fixed at some point.

path: pass `path` to `peer` in `hg incoming` We directly use the `path` object to build the `peer` object. We don't use it for sub-repositories yet, which will have to be fixed at some point.

path: pass `path` to `peer` in `hg incoming` bookmark logic We directly use the `path` object to build the `peer` object. Touching this code highlighted that we never honor the branches' information of when doing bookmark level incoming checks.

path: remove outdated documentation point from `get_unique_pull_path` This is no longer true.

path: update `get_unique_pull_path` to point out it returns a url Unlike other functions in this module it returns a url as `bytes` and not a `path` object. Let us point it out in the doc.

changelog-v2: fix the docket `struct` The previous definition used `L` which is actually 4 bytes, while the documentation and intend was to use `Q`, i.e. 8 bytes.

path: pass `path` to `peer` in infinite push We directly use the `path` object to build the `peer` object.

path: pass `path` to `peer` in `hg histedit` We directly use the `path` object to build the `peer` object.

path: pass `path` to `peer` in the `outgoing` revset We directly use the `path` object to build the `peer` object.

path: pass `path` to `peer` in `hg summary` We directly use the `path` object to build the `peer` object.

path: pass `path` to `peer` in `hg outgoing` We directly use the `path` object to build the `peer` object.

path: pass `path` to `peer` in `hg bundle` We directly use the `path` object to build the `peer` object.

path: have `peer` constructor accept a `path` object We don't do anything fancy with it yet, but this is an important step towards having the peers aware of their "source" and associated configurations.

path: deprecated the `pushloc` attribute We want to make sure people with use the full featured path "variant".

path: update logic in `perf` to use the push variant when available The command seems currently broken, but at least it won't be broken by us !

path: directly use the push_variant in `infinitepush` We don't need any extra processing now.

path: directly use the push_variant in `hg histedit` outgoing logic We don't need any extra processing now.

path: directly use the push_variant in the `outgoing` revset We don't need any extra processing now.

path: directly use the push_variant in outgoing internals We don't need any extra processing now.

path: directly use the push_variant in `hg summary` We don't need any extra processing now.

path: directly use the push_variant in `hg outgoing` We don't need any extra processing now.

path: directly use the push_variant in `hg push` We don't need any extra processing now.

path: have `get_push_paths` directly return the push variants So the function directly returns usable paths, that should help the callers!

path: add a method to retrieve a "push variant" of a path This gets you the same path, but using the `pushurl` as destination. This opens the way for a lot of different improvements, the one which interests us is having `peer` objects aware of the `path` they came from.

path: move the url parsing and related attribute setting to a method This will make it simpler to reuse this logic in the next changeset.

peer-or-repo: remove the now unused function We do not need it anymore.

peer-or-repo: build a repo directly in the `repo` function We skip the ambiguous _peerorrepo function to explicitly build a repo within the dedicated function. The peer scheme are therefore no longer considered to build the object.

peer-or-repo: build a peer directly in the `peer` function We skip the ambiguous _peerorrepo function to explicitly build a peer within the dedicated function. This mean explicitly getting a peer from an explicitly create repository when necessary.

static-http: have `statichttprepo.instance` return a peer object It previously returned a statichttprepo object which could not be used for anything but creating a peer. So lets put it into the peer bucket with a peer behavior.

scheme: move the drive letter checking in its own function This help the readability of the main function as is was taking much more room than the main logic.

peer-or-repo: split the scheme between repo and peer Some of the scheme will always produce a peer and some will always produce a repository. So lets use different mapping to reduce the ambiguity.

peer-or-repo: stop relying on AttributeError in `islocal` This will confused pytypes in a future changeset.

repo-or-peer: deprecate calling `islocal` on non-path object There object have a `.local()` method and should use it.

share: stop using 'islocal' with repo instance Having this level of polymorphism of the `islocal` function is weird, and we already have a way to know if the repo is local from the object itself. We are about to deprecate passing a non-bytes object to `islocal`, so clean this up beforehand. We might want to clean up this function too in the future, however this is another adventure.

peer-or-repo: make sure object in "scheme" have a `instance` object The previous form of having heterogeneous object in the dictionnary makes things more complicated than they needed to be. I am not super happy about the current (especially around 'islocal', that most item do not have), but this is already much better.

peer-or-repo: move the object setup in its own function The `_peerorrepo` function is problematic, because it can build different types of object (repository and peer). This make it hard to adjust the arguments to the type of object we needs. So this patch start a series of change to create peer and repo without going through a common function. We move the part of the function doing object setup it its own function to make it simpler to reuse in others contexts.

tests: use an all too familiar executable in test-run-tests.t (issue6661) true(1) sometimes lives in /usr/bin/, and so this test fails on such systems. We also can't use which(1), because it's apparently not POSIX and Debian complains about it [1]. We also cannot really create a script and chmod +x it, because this is a symlink case, execbit case is slightly below. So let's use something that we know is executable, but not hg itself.

tests: add the missing space to test-hghave.t (issue6762) log() in run-tests.py add a space to the end of all messages that it prints, and I missed this fact in my previous patch.

delta-find: use a single snapshot cache when applying a group to an object This will avoid walking the revlog over and over again in some situations. The difference is hard to show in our current benchmark suite, as the gain is lower than their overall instability.

delta-find: make sure we only use newer full snapshot as candidate The current code does not needs to protect against this, as there are no older snapshot in the current cache. However as we are getting ready to reuse this cache from one revision to another, we need the code to protect itself about what's coming.

delta-find: use sets instead of list in the snapshot cache This seems more appropriate.

delta-find: use a smarter object for snapshot caching This open the way for a longer lived cache.

find-delta: pass the cache-delta usage policy alongside the cache-delta The idea is to give higher level code more control to what will happens with the cache delta passed. This should help with controling how we treat delta's from different sources. The final goal of this change is to allow for server modes where the client can blindly accept any server delta without regards to any local constraints. This will be implemented in later changesets.

find-delta: move most of the debug-find-delta code in the debug module Lets us that module more. It will help us to keep revlog implementation details close to each other.

find-delta: minor preparatory change We are about to add more item in the cachedelta object, so lets access the item by index instead of doing a full extensions.

find-delta: rename _isgooddeltainfo Lets move to a more readable name now that we are allowed to. This cannot hurt.

test-revlog-raw: drop the overwrite of dead code The revlog class no longer have a _isgooddeltainfo method for a long time. So overwriting it does not get us anything. The test have been wrapping the right code since then anyway.

emitrevision: consider ancestors revision to emit as available base This should make more delta base valid. This notably affects: * case where we skipped some parent with empty delta to directly delta against an ancestors * case where an intermediate snapshots is stored. This change means we could sent largish intermediate snapshots over the wire. However this is actually a sub goal here. Sending snapshots over the wire means the client have a high odd of simply storing the pre-computed delta instead of doing a lengthy process that will… end up doing the same intermediate snapshot. In addition the overall size of snapshot (or any level) is "only" some or the overall delta size. (0.17% for my mercurial clone, 20% for my clone of Mozilla try). So Sending them other the wire is unlikely to change large impact on the bandwidth used. If we decide that minimising the bandwidth is an explicit goal, we should introduce new logic to filter-out snapshot as delta. The current code has no notion explicite of snapshot so far, they just tended to fall into the wobbly filtering options. In some cases, this patch can yield large improvement to the bundling time: ### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog # benchmark.name = perf-bundle # benchmark.variants.revs = last-100000 before: 68.787066 seconds after: 47.552677 seconds (-30.87%) That translate to large improvement to the pull time : ### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog # benchmark.name = pull # benchmark.variants.issue6528 = disabled # benchmark.variants.revs = last-100000 before: 142.186625 seconds after: 75.897745 seconds (-46.62%) No significant negative impact have been observed.

sqlitestore: add an `ancestors` method We will need it during bundling. The implementation mirror the one in revlog.

emitrevision: if we need to compute a delta on the fly, try p1 or p2 first Falling back to `prev` does not yield any real value on modern storage and result in pathological changes to be created on the other side. Doing a delta against a parent will likely be smaller (helping the network) and will be safer to apply on the client (helping future pulls by Triggering intermediate snapshop where they will be needed by later deltas).

emitrevision: simplify the fallback to computed delta Not using the stored delta, or having a full snapshot on disk behave the same ways, so lets use the same code path for that, this is simpler, and it update will be simpler.

emitrevision: also check the parents in the availability closure One of the point of having a closure is to gather the logic in it. So we gather the logic. The `parents[:]` part is a bit ugly but will be replaced by better code soon anyway.

emitrevision: add a small closure to check if a base is usable We will make more use of this and make it more complex too.

reuse-delta-base: improves some documentation The current code got me a bit confused initially. So a bit more documentation around it cannot hurt.

tests: expect the message from 1baf0fffd82f in test-hghave.t (issue6762) Couldn't reproduce locally, but apparently this message may occasionally pop up when running this test.

branching: merge stable into default

changelog-v2: add a configuration to disable rank computation We encountered a graph where rank computation was pathologically slow. We add an option to disable this computation while this is getting fixed. Disabling the rank computation should allow for testing other changelog-v2 features undisturbed (like changeset-based copy tracing). I am purposely not adding a test for the new non-default code path, as this is a temporary work around of an experimental feature.

debugrevlog: display total stored information This is an interesting statistis, so let's display it.

debug-revlog: details about non-ancestors delta-bases Deltas against a base that is not an ancestor of the revision that owns this delta are notable. For example, they introduce complexity during the bundling process as the base might not exist on the unbundling side. We detect them in `hg debugrevlog` and print information about them.

debug-revlog: move the code in revlogutils module We have a module dedicated to debug code, let us use it.

debug-revlog: move the --dump code in `revlogutils` module We have a module dedicated to debug code, let us use it.

delta-find: set the default candidate chunk size to 10 I ran performance and storage tests on repositories of various sizes and shapes for the following values of the config : 5, 10, 20, 50, 100, no-chunking The performance tests do not show any statistical impact on computation times for large pushes and pulls. For searching for an individual delta, this can provide a significant performance improvement with a minor degradation of space-quality on the result. (see data at the end of the commit). For overall store size, the change : - does not have any impact on many small repositories, - has an observable, but very negligible impact on most larger repositories. - One private repository we use for testing sees a small increase in size (1%) in the narrower version. We will try to get more numbers on a larger version of that repository to make sure nothing pathological happens. We pick "10" as the limit as "5" seems a bit more risky. There are room to improve the current code, by using more aggressive filtering and better (i.e any) sorting of the candidates. However this is already a large improvement for pathological cases, with little impact in the common situations. The initial motivation for this change is to fix performance of delta computation for a file where the previous code ended up testing 20 000 possible candidate-bases in one go, which is… slow. This affected about ½ of the file revisions leading to atrocious performance, especially during some push/pull operations. Details about individual delta finding timing: ---------------------------------------------- The vast majority of benchmark cases are unchanged but the three below. The first two do not see any impact on the final delta. The last one sees a change in delta-size that is negligible compared to the full text size. ### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog # benchmark.name = perf-delta-find # benchmark.variants.rev = manifest-snapshot-many-tries-a (revision 756096) ∞: 5.844783 5: 4.473523 (-23.46%) 10: 4.970053 (-14.97%) 20: 5.770386 (-1.27%) 50 5.821358 100: 5.834887 MANIFESTLOG: rev = 756096: (no-limit) delta-base = 301840 search-rounds = 6 try-count = 60 delta-type = snapshot snap-depth = 7 delta-size = 179 MANIFESTLOG: rev=756096: (limit = 10) delta-base=301840 search-rounds=9 try-count=51 delta-type=snapshot snap-depth=7 delta-size=179 ### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog # benchmark.name = perf-delta-find # benchmark.variants.rev = manifest-snapshot-many-tries-d (revision 754060) ∞: 5.017663 5: 3.655931 (-27.14%) 10: 4.095436 (-18.38%) 20: 4.828949 (-3.76%) 50 4.987574 100: 4.994889 MANIFESTLOG: rev=754060: (no limit) delta-base=301840 search-rounds=5 try-count=53 delta-type=snapshot snap-depth=7 delta-size = 179 MANIFESTLOG: rev=754060: (limite = 10) delta-base=301840 search-rounds=8 try-count=45 delta-type=snapshot snap-depth=7 delta-size = 179 ### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog # benchmark.name = perf-delta-find # bin-env-vars.hg.flavor = rust # benchmark.variants.rev = manifest-snapshot-many-tries-e (revision 693368) ∞: 4.869282 5: 2.039732 (-58.11%) 10: 2.413537 (-50.43%) 20: 4.449639 (-8.62%) 50 4.865863 100: 4.882649 MANIFESTLOG: rev=693368: delta-base=693336 search-rounds=6 try-count=53 delta-type=snapshot snap-depth=6 full-test-size=131065 delta-size=199 MANIFESTLOG: rev=693368: delta-base=278023 search-rounds=5 try-count=21 delta-type=snapshot snap-depth=4 full-test-size=131065 delta-size=278 Raw data for store size (in bytes) for various chunk size value below: ---------------------------------------------------------------------- 440 134 384 5 pypy/.hg/store/ 440 134 384 10 pypy/.hg/store/ 440 134 384 20 pypy/.hg/store/ 440 134 384 50 pypy/.hg/store/ 440 134 384 100 pypy/.hg/store/ 440 134 384 ... pypy/.hg/store/ 666 987 471 5 netbsd-xsrc-2022-11-15/.hg/store/ 666 987 471 10 netbsd-xsrc-2022-11-15/.hg/store/ 666 987 471 20 netbsd-xsrc-2022-11-15/.hg/store/ 666 987 471 50 netbsd-xsrc-2022-11-15/.hg/store/ 666 987 471 100 netbsd-xsrc-2022-11-15/.hg/store/ 666 987 471 ... netbsd-xsrc-2022-11-15/.hg/store/ 852 844 884 5 netbsd-pkgsrc-2022-11-15/.hg/store/ 852 844 884 10 netbsd-pkgsrc-2022-11-15/.hg/store/ 852 844 884 20 netbsd-pkgsrc-2022-11-15/.hg/store/ 852 844 884 50 netbsd-pkgsrc-2022-11-15/.hg/store/ 852 844 884 100 netbsd-pkgsrc-2022-11-15/.hg/store/ 852 844 884 ... netbsd-pkgsrc-2022-11-15/.hg/store/ 1 504 227 981 5 netbeans-2018-08-01-sparse-zstd/.hg/store/ 1 504 227 871 10 netbeans-2018-08-01-sparse-zstd/.hg/store/ 1 504 227 813 20 netbeans-2018-08-01-sparse-zstd/.hg/store/ 1 504 227 813 50 netbeans-2018-08-01-sparse-zstd/.hg/store/ 1 504 227 813 100 netbeans-2018-08-01-sparse-zstd/.hg/store/ 1 504 227 813 ... netbeans-2018-08-01-sparse-zstd/.hg/store/ 3 875 801 068 5 netbsd-src-2022-11-15/.hg/store/ 3 875 696 767 10 netbsd-src-2022-11-15/.hg/store/ 3 875 696 757 20 netbsd-src-2022-11-15/.hg/store/ 3 875 696 653 50 netbsd-src-2022-11-15/.hg/store/ 3 875 696 653 100 netbsd-src-2022-11-15/.hg/store/ 3 875 696 653 ... netbsd-src-2022-11-15/.hg/store/ 4 531 441 314 5 mozilla-central/.hg/store/ 4 531 435 157 10 mozilla-central/.hg/store/ 4 531 432 045 20 mozilla-central/.hg/store/ 4 531 429 119 50 mozilla-central/.hg/store/ 4 531 429 119 100 mozilla-central/.hg/store/ 4 531 429 119 ... mozilla-central/.hg/store/ 4 875 861 390 5 mozilla-unified/.hg/store/ 4 875 855 155 10 mozilla-unified/.hg/store/ 4 875 852 027 20 mozilla-unified/.hg/store/ 4 875 848 851 50 mozilla-unified/.hg/store/ 4 875 848 851 100 mozilla-unified/.hg/store/ 4 875 848 851 ... mozilla-unified/.hg/store/ 11 498 764 601 5 mozilla-try/.hg/store/ 11 497 968 858 10 mozilla-try/.hg/store/ 11 497 958 730 20 mozilla-try/.hg/store/ 11 497 927 156 50 mozilla-try/.hg/store/ 11 497 925 963 100 mozilla-try/.hg/store/ 11 497 923 428 ... mozilla-try/.hg/store/ 10 047 914 031 5 private-repo 9 969 132 101 10 private-repo 9 944 745 015 20 private-repo 9 939 756 703 50 private-repo 9 939 833 016 100 private-repo 9 939 822 035 ... private-repo

delta-find: add a way to control the number of bases tested at the same time See inline comment for details. The feature is currently disabled, but should be enabled by default to mitigate some existing pathological cases. Also see the next changeset for details.

test: adjust test-push-race.t timeout's to overall test timeout The generic `tests/testlib/wait-on-file` mechanism scale its timeout with the value of `HGTEST_TIMEOUT`, the `delaypush.py` in `test-push-race.t` is not doing this, and we have been seeing more and more timeout from loaded CI worker lately. Adding this timeout scaling should help with that.

setup: include vendored 3rd party type stubs While pytype may not support PEP 561, PyCharm does, so having the stubs available means it can determine `foo = attr.ib(type=int)` means `foo` is an int. This only applies when using Mercurial as a library, like with TortoiseHg development- PyCharm is already smart enough to use the *.pyi files in the Mercurial source tree when hacking on Mercurial itself. I left the mercurial.cext stubs out because it seems very low level, that 3rd parties shouldn't be using directly.

ci: bump pytype to 2022.11.18 No particular reason, other than the current build is fairly old. It flagged a few more things (that weren't errors based on the logic around them), but OTOH, some of the pyi stubs it generates are less specific.

ci: run the script to add vendored type stubs to typeshed Since CI runs from docker images, it doesn't matter that this mucks with the typeshed bundled with pytype.

contrib: add a script for adding vendored type stubs to typeshed I really hate this, but pytype doesn't support PEP 561 and doesn't seem to have the equivalent of `MYPYPATH` to point to custom stubs. Ignoring the vendored stubs isn't necessarily harmful, but pytype has been choking on the vendored attr package after pytype 2022.03.29 with errors like this: File "/mnt/c/Users/Matt/hg/mercurial/linelog.py", line 52, in __iter__: Built-in function iter was called with the wrong arguments [wrong-arg-types] Expected: (collection: bytearray) Actually passed: (collection: mercurial.thirdparty.attr._make._CountingAttr) File "/mnt/c/Users/Matt/hg/mercurial/dirstateutils/v2.py", line 143, in pack: Built-in function len was called with the wrong arguments [wrong-arg-types] Expected: (obj: Sized) Actually passed: (obj: mercurial.thirdparty.attr._make._CountingAttr) Attributes of protocol Sized are not implemented on mercurial.thirdparty.attr._make._CountingAttr: __len__ File "/mnt/c/Users/Matt/hg/mercurial/dirstateutils/v2.py", line 144, in pack: No attribute 'rfind' on mercurial.thirdparty.attr._make._CountingAttr [attribute-error] File "/mnt/c/Users/Matt/hg/mercurial/dirstateutils/v2.py", line 146, in pack: Built-in function len was called with the wrong arguments [wrong-arg-types] Expected: (obj: Sized) Actually passed: (obj: mercurial.thirdparty.attr._make._CountingAttr) Attributes of protocol Sized are not implemented on mercurial.thirdparty.attr._make._CountingAttr: __len__ File "/mnt/c/Users/Matt/hg/mercurial/dirstateutils/v2.py", line 152, in pack: No attribute 'v2_data' on mercurial.thirdparty.attr._make._CountingAttr [attribute-error] File "/mnt/c/Users/Matt/hg/mercurial/util.py", line 2817, in go: unsupported operand type(s) for /: 'count: mercurial.thirdparty.attr._make._CountingAttr' and 'float: float' [unsupported-operands] No attribute '__truediv__' on 'count: mercurial.thirdparty.attr._make._CountingAttr' or '__rtruediv__' on 'float: float' Called from (traceback): line 2981, in __bytes__ This is essentially the same hack we've been using in TortoiseHg to add the vendored PyQt5 stubs. What I don't understand is pytype *still* generates *.pyi files under .pytype/pyi/mercurial/thirdparty/attr, even when the package is explicitly ignored in the pytype command line args. But it avoids the errors, which means we aren't stuck on pytype==2022.03.29. https://github.com/google/pytype/issues/151

contrib: update check-pytype.sh to list stubs that caused pytype to crash The same logic is in the TortoiseHg tests for running pytype, and it's useful to know if a new version of pytype is better or worse.

typing: add py.typed to mercurial.cext for PEP 561 support Unfortunately, pytype doesn't support this yet. But it was included with the attr package, so we might as well do it here for consistency. Unlike the attr package, these type hints are only partial, so they are marked as such[1] (but who knows if it matters, given these are C extensions, so no local source code to scan). [1] https://peps.python.org/pep-0561/#partial-stub-packages

typing: add missing signature for mercurial.cext.parsers.parse_index2() Flagged by pytype 2022.11.18 when the cext stubs are made visible to it. There are very likely other signatures that are missing, but this is enough to keep it happy for now.

typing: minor tweaks to allow updating to pytype 2022.11.18

python-compat: adapt to Python 3.11 BC breakage with `random.sample` As per https://docs.python.org/3/whatsnew/3.11.html#porting-to-python-3-11: "The population parameter of `random.sample()` must be a sequence, and automatic conversion of sets to lists is no longer supported. Also, if the sample size is larger than the population size, a `ValueError` is raised"

typing: add type hints to mercurial/help.py Was hoping to find more issues like f09bc2ed9100, but it may be that nothing checks the args to that operation. In any event, the work is done and pytype doesn't do a very good job inferring the types. A few of th emore complicated things like the command table are left untyped, because they come from modules that aren't typed yet.

match: make the FLAG_RE pattern a raw string PyCharm was complaining about invalid escape sequences since this was added recently in 3eda36e9b3d6.

configitems: add a default value for "merge-tools.xxx.regappend" When trying to figure out how `hg help -v` took the Set interpolation path in f09bc2ed9100, I turned on devel warnings and noticed this (unrelated) warning: devel-warn: specifying a mismatched default value for a registered config item: 'merge-tools.beyondcompare4.regappend' '' at: c:\Users\Matt\hg\mercurial\filemerge.py:46 (_toolstr) The previous default value for this config was `None`, but that slightly complicates the code at the only site it is used, referenced above.

attr: vendor 22.1.0 The previous version was 5 years old, and pytype 2022.06.30 started complaining about various uses (e.g. seeing `mercurial.thirdparty.attr._make._CountingAttr` instead of `bytearray`). Hopefully this helps. Additionally, this has official python 3.11 support. The `attrs` package is left out, because it is simply a bunch of *.pyi stubs and `from attr.X import *`, and that's not how they've been used up to this point. We'd probably need to customize those anyway to `from mercurial.thirdparty.attr import *`.

tests: update test-util.py for modern attrs package When updating to 22.1.0, this test started failing: Traceback (most recent call last): File "/tmp/mercurial-ci/tests/test-util.py", line 53, in <module> _start_default = (util.timedcmstats.start.default, 'factory') AttributeError: type object 'timedcmstats' has no attribute 'start' Poking around in `hg debugshell`, the attribute is indeed missing, but looks to be attached to `__attrs_attrs__` in both the currently vendored and the modern version of attrs. The old attrs packages will print the same for both accesses, so fingers crossed... >>> print((util.timedcmstats.start.default, 'factory')) (Factory(factory=<function timedcmstats.<lambda> at 0x000001EFDF0F21F0>, takes_self=False), 'factory') >>> print((util.timedcmstats.__attrs_attrs__.start.default, 'factory')) (Factory(factory=<function timedcmstats.<lambda> at 0x000001EFDF0F21F0>, takes_self=False), 'factory')

help: fix a py3 error interpolating Set into b'%s' I can't reproduce it, but a coworker hit this with `hg help -v` with 6.2.3: ... File "mercurial\help.pyc", line 865, in helplist TypeError: %b requires a bytes-like object, or an object that implements __bytes__, not 'set' I can confirm that the original expression fails in `hg debugshell`, and the new one works. The second instance was found by searching for "%s", but PyCharm detects a lot of variables as Any type, so I have no idea if there are other lurking problems.

rhg: upgrade the remainder of the dependencies These are painless, so they are all grouped in this changeset.

rhg: upgrade `clap` dependency This one is the worst one to upgrade since v2 -> v4 broke a ton of API, which thankfully seems saner now. Contrary to what was done in the `hg-core/src/examples/nodemap` rewrite, we're not switching from the "builder" pattern to the "derive" pattern, since that would imply a much larger diff. It can be done incrementally.

hg-cpython: upgrade dependencies `hg-cpython` has no BC breaking dependencies, we can group them all in this changeset.

hg-core: upgrade all remaining dependencies Finally, these dependencies do not require any code changes, so they are all grouped in the same changeset.

hg-core: upgrade `clap` dependency Upgrading is a finally possible now that we're supporting a more recent version of Rust. This required changes in the `nodemap` example.

hg-core: upgrade `zstd` dependency Now that we support a newer version of Rust, we can update this dependency to get all the latest bugfixes and improvements. A slight API adjustment was needed.

hg-core: make use of `strip_suffix` now that we're using Rust 1.45+

rust: use `matches!` macro now that we're using Rust 1.42+

hg-core: remove unneeded util now that we support Rust 1.42+

hg-core: remove unneeded trait now that we support Rust 1.52+

rust: remove newly redundant `use` statements with the 2021 edition prelude https://doc.rust-lang.org/edition-guide/rust-2021/prelude.html

rust: move all crates in the main workspace to edition 2021 We've changed our minimum Rust version to 1.61.0 in the previous patch, and edition 2021 predates that version.

rust: upgrade supported Rust toolchain version A few months ago¹, a decision was made to move the Rust toolchain target to whatever Debian Testing was tracking. I didn't have the bandwidth to act on it until now. This is starting to be even more problematic than before, now that edition 2021 is out. The CI has been updated to track the current Debian testing version, 1.61.0. [1] https://lists.mercurial-scm.org/pipermail/mercurial-packaging/2022-April/000338.html

branching: merge stable into default

tests: fix test-sparse-revlog This one is not covered by the CIbecause I requires an expensive artifact to be cached. So it goes out of think on regular basis (we should fix that…) The test ouput was affected by e706bb41fdb3 as we filtering now happens sooner, removing for the output.

Added signature for changeset c890d8b8bc59

Added tag 6.3.1 for changeset c890d8b8bc59

relnotes: add 6.3.1

memory-usage: fix `hg log --follow --rev R F` space complexity When running `hg log --follow --rev REVS FILES`, the log code will walk the history of all FILES starting from the file revisions that exists in each REVS. Before doing so, it looks if the files actually exists in the target revisions. To do so, it opens the manifest of each revision in REVS to look up if we find the associated items in FILES. Before this changeset this was done in a way that created a changectx for each target revision, keeping them in memory while we look into each file. If the set of REVS is large, this means keeping the manifest for each entry in REVS in memory. That can be large… if REV is in the form `::X`, this can quickly become huge and saturate the memory. We have seen usage allocating 2GB per second until memory runs out. So this changeset invert the two loop so that only one revision is kept in memory during the operation. This solve the memory explosion issue.

tests: run many tests in $TESTTMP/repo instead of $TESTTMP This is useful so we can store other files in $TESTTMP (in particular tests that use docket files (nodemap, dirstate-v2) keep file uids in $TESTTMP/UID)

tests: fix the detection of dirstate-v2 in hghave.py

dirstate-v2: do not put the dirstate data file in a transaction, since the transaction reverts the store, while the dirstate is stored separately

commit: properly consider file include and exclude options when closing branch It looks like this is meant to prevent adding another commit that does nothing but close a branch on top of a commit that already closed the branch. The matcher building functions want `Dict[bytes, Any]`, not `Dict[str, Any]`, which was found by adding type hints to the matcher related methods in scmutil.

tests: demonstrate a bug blocking a redundant branch close

delta-find: adjust the moment when we mark something as "tested" In a coming change, not all elements of `group` might get tested. So we need to have more control about when a revision is actually added to the `tested` set. So we move to a more verbose (and more fragile) version.

delta-find: rename a variable for clarity the index in the delta-chain is also the snapshot depth. So we rename the variable for clarity.

delta-find: small documentation update This is not a 1-1 mapping, but a 1-n mapping. Lets make the associated comment clearer.

delta-find: move pre-filtering with other pre-filtering logic This is more consistent and will help use to be in a clean state before dealing with the "too large group" issue. As a side effect, the debug output now skip some useless cases, making it more useful.

delta-find: expand a function definition and call before extendin it This make the next changeset more compact.

debug: add an option to display statistic about a unbundling operation This will helps a lot to understand how the bundling decision have actually impacted pull/unbundle on the other side.

debug: add an option to display statistic about a bundling operation This will helps a lot to understand how the bundling decision might impact pull/unbundle on the other side.

delta-find: add debug information about reuse of cached data This will help us to understand the behavior of find-delta during a pull.

tests: stop creating temporary files in TESTDIR

cffi: fix a bytes vs str issue on macOS when listing directories This code hasn't been touched in recent years, and the other implementation return bytes for the filename, so I assume this is a holdover from the py2 days. I was unable to test it on mac though, because the `_osutil` import failed.

packaging: refresh dependency hashes (issue6750) Also, add some documentation to the `.in` files.

matcher: do not prepend '.*' to pattern using ^ after flags Since the previous commit (fixing wider issue), the code generated strange regex. This is now fixed and tested.

matcher: fix the issue with regex inline-flag in rust oo Same problem same solution.

matcher: fix issues regex flag contained in pattern (issue6759) Python 3.11 is now enforcing that flag must be at the beginning of the regex This creates a serious regression for people using Python 3.11 with an hgignore using flag in a "relre" pattern. We now detect any flags in such pattern and "prepend" our ".*" pattern after them. In addition, we now insert the flag in the regexp to only affect the pattern we are rewriting. Otherwise, the regex built from the combined pattern would these flags in the middle of it anyway. As a side effect of this last change, we fix a bug… before this change regex flag in a pattern would affect all combined patterns. That was bad and is not longer the case. The Rust code needs to be updated to fix that very bug, but we will do it in another changeset.

release: removed the 6.3.0 tag The revision having two tags confuses some of the build script trying to fetch the version. So, remove the "bad" tag for now.

cffi: fix a bytes vs str issue on macOS when listing directories This code hasn't been touched in recent years, and the other implementation return bytes for the filename, so I assume this is a holdover from the py2 days. I was unable to test it on mac though, because the `_osutil` import failed.

typing: fix the typehint for `skip` arg on `osutil.listdir()` to be optional The cffi and pure implementations both default to `None`, and the args are parsed in C with `"y#|OO:listdir"`, with the C variable initialized to NULL.

typing: add type hints to mpatch implementations Again, using `merge-pyi` to apply the stubs in cext and then manually type the private methods. The generated stub without these hints inferred very little, and the stuff it did was wrong.

typing: add type hints to bdiff implementations Not super important code, but this was an exercise in using `merge-pyi` to fold type stubs back into the code on something small. The cext stubs don't seem to be getting used (at least the only thing in `.pytype/pyi/mercurial/cext` after a run generating the stubs is `__init__.pyi`), so maybe this will help some.

cffi: adjust the list returned by bdiff.blocks to never have a None entry This was flagged by pytype after merging the corresponding bdiff.pyi in cext: File ".../mercurial/cffi/bdiff.py", line 44, in blocks: bad return type [bad-return-type] Expected: List[Tuple[int, int, int, int]] Actually returned: List[None] AFAICT, all callers immediately unpack the tuple into 4 variables, so a `None` entry would simply crash if they aren't all overwritten. As long a `count` and the link list are consistent, this shouldn't be a problem. This placates both pytype and PyCharm (which complained about the `i` in `rl[i]` having the wrong type with the old code).

typing: fix a syntax error in mercurial/cext/bdiff.pyi I noticed because `merge-pyi` on the non-cext implementations with this file as input skipped the return type for this. `pytype-single --parse-pyi` confirmed it was a problem.

hg-core: relax dependencies pinning Being this rigid makes packagers' job more difficult since they might not carry the exact version. This hard pinning was introduced in eb02decdf but wasn't strictly necessary to achieve its compatibility goal.

tests: make running ls in a no longer existing directory more portable On Linux, ls -A simply returns nothing and the exit code is 0. On NetBSD, ls -A complains that . doesn't exist and the exit code is 1. Sadly, it's not possible to do something like "[1] (?)", so " || true" is the best I could come up with.

tests: move some lines inside #if windows-#else block test-removeemptydirs.t This is done to make it's clear that windows is not affected by this test case, IOW windows and non-windows platforms are now tested separately, because their results are very different.

tests: use ls -A instead of ls -1 in test-removeemptydirs.t In case the tests are run as root, ls assumes -A by default on some systems (e.g. NetBSD). Tests probably shouldn't be run as root, but let's use -A just in case, for convenience.

release: add the "6.3" to the 6.3 release The release was tagged "6.3.0" while the custom have been to omit the third ".0" for initial release for the history of the project. For consistency, we add the customary version of the tag to the 6.3 release.

tests: fix new git protocol policy in convert-git.t Recent versions of git restrict the use of the "file" protocol for security reasons (https://github.com/git/git/commit/a1d4f67). This broke this test, which failed with the following hidden error: "fatal: transport 'file' not allowed". This patch relaxes the git configuration for the test to solve this.

debug-delta-find: add a --source option This will help us to understand the delta-find operation in different situations.

demandimport: ensure lazyloaderex sets loader attributes (issue6725) Adds test capturing missed expectation.

shelve: add test for Shelf.changed_files

shelve: add Shelf.changed_files for resolving changed files in a plugin

branching: merge stable into default

relnotes: add final 6.3 relnotes

Added signature for changeset 04f1dba53c96

Added tag 6.3.0 for changeset 04f1dba53c96

rust: create wrapper struct to reduce `regex` contention issues Explanations inline.

tags-fnode-cache: skip building a changectx in getfnode Building a changectx object is costly, doing it just to retrieve the revision number is suboptimal. Directly fetching the revision number from the changelog provide a sizeable speedup to `hg debugupdatecache`. ### data-env-vars.name = mercurial-2018-08-01-zstd-sparse-revlog # benchmark.name = debug-update-cache # benchmark.variants.pre-state = warm before: 0.213229 seconds after: 0.165577 seconds (-22.35%) # data-env-vars.name = mercurial-filtered-2019-11-22-zstd-sparse-revlog before: 1.200383 seconds after: 1.071618 seconds (-10.73%) # data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog before: 1.465735 seconds after: 0.923128 seconds (-37.02%) # data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog before: 6.511771 seconds after: 4.507316 seconds (-30.78%) # data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog before: 1.023007 seconds after: 0.645026 seconds (-36.95%) # data-env-vars.name = pypy-2018-08-01-zstd-sparse-revlog before: 0.381141 seconds after: 0.268654 seconds (-29.51%)

tags-fnode-cache: do not repeatedly open the filelog in a loop While getting multiple hgtagsfnodecache entries, we were opening (and closing) the `.hgtags` filelog for each iteration. The meant repeatedly reading and parsing the version same information from disk. A quite costly operation. We no longer do this, leading to a sizable improvement in `hg debugupdatecache` run for an already warm repositories. ### data-env-vars.name = mercurial-2018-08-01-zstd-sparse-revlog # benchmark.name = debug-update-cache # benchmark.variants.pre-state = warm before: 1.711778 seconds after: 0.213229 seconds (-87.54%) # data-env-vars.name = pypy-2018-08-01-zstd-sparse-revlog before: 4.010817 seconds after: 0.381141 seconds (-90.50%) # data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog before: 13.574141 after: 1.023007 seconds (-92.46%) # data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog before: 18.884656 after: 1.465735 seconds (-92.24%) # data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog before: 88.924823 after: 6.511771 seconds (-92.68%)

profile: prevent a crash when line number is unknown For some unknown reason, `self.lineno` can be None. The previous code crashed in such case, we now ignore the case, as we do for other error in this function. We also fallback to using `-1` in the output when this lack of line number makes it to the display code. The reason of unknown line-numbers is… unknown.

configitems: enable pullbundles by default The use of pullbundles is already protected: they are only used when a pullbundles.manifest file is created on the server. Having an additional flag doesn't really make sense and can confuse users (as indicated in the mercurial mailing list on the topic "Can't get pull-bundles working").

upgrade: byteify requirement changes written to output ui.write() expects bytes, and internally uses `b''` as the default when getting the `label` keyword from `*args`. So either we're missing test coverage, or there's some very subtle conversion from unicode somewhere. Also, slip in a type hint to flag this in the future.

typing: add basic type hints to stringutil.py

vfs: make the default opener mode binary The default was already binary for `abstractvfs`, and the `vfs` implementation adds binary mode if the caller didn't supply it. Therefore, it should be safe for all vfs objects (and I don't think we want text reads anyway).

typing: add basic type hints to vfs.py Again, there's a lot more that could be done, but this sticks to the obviously correct stuff that is related to primitives or `vfs` objects. Hopefully this helps smoke out more path related bytes vs str issues in TortoiseHg. PyCharm seems smart enough to apply hints from annotated superclass functions, but pytype isn't (according to the *.pyi file generated), so those are annotated too. There was some discussion about changing the default path arg from `None` to `b''` in order to avoid the more verbose `Optional` declarations. This would be more in line with `os.path.join()` (which rejects `None`, but ignores empty strings), and still not change the behavior for callers still passing `None` (because the check is `if path` instead of an explicit check for `None`). But I didn't want to hold this up while discussing that, so this documents what _is_.

util: implement `writelines()` on atomictempfile With typehints on the vfs objects, pytype will flag this: FAILED: /mnt/c/Users/Matt/hg/.pytype/pyi/mercurial/patch.pyi /usr/bin/python3.8 -m pytype.single --imports_info /mnt/c/Users/Matt/hg/.pytype/imports/mercurial.patch.imports --module-name mercurial.patch -V 3.7 -o /mnt/c/Users/Matt/hg/.pytype/pyi/mercurial/patch.pyi --analyze-annotated --nofail --quick /mnt/c/Users/Matt/hg/mercurial/patch.py File "/mnt/c/Users/Matt/hg/mercurial/patch.py", line 535, in writerej: No attribute 'writelines' on mercurial.util.atomictempfile [attribute-error] In Union[ mercurial.util.atomictempfile, mercurial.vfs.checkambigatclosing, mercurial.vfs.delayclosedfile, mercurial.windows.fdproxy, mercurial.windows.mixedfilemodewrapper ] It's not a real problem there (atomictempfile is only created by passing different args), but it's reasonable for this to implement the function and behave like a normal file. There are other functions missing that can be added if/when needed.

typing: add basic type hints to localrepo.py There's a lot more that could be done, but this sticks to the obviously correct stuff that is either related to existing imports or primitives. Hopefully this helps smoke out more path related bytes vs str issues in TortoiseHg. I'm avoiding the interfaces for now, because they seem to confuse pytype and/or PyCharm. It might be worth typing the return of `makelocalrepository` to `localrepository`, but that leaks an implementation detail, so that can be revisited later.

check-code: drop the check for whitespace around named parameters This check flags py3 annotations of named parameters, because `black` adds spaces around the assignment in this case. Since the chosen formatter has opinions (and pylint also wants the space in the case of annotations), drop the check so we can use py3 annotations.

rhg: add a config option to fall back immediately This is useful for debugging the behavior of the "default" `hg` in tests without having to manually substitute the fallback path.

rhg: stop shadowing `exit` function This will be useful for the next patch which needs it.

config: add alias from `hg help rhg` to `hg help rust` This will make using `rhg` more user-friendly and features more discoverable.

rhg: add `config.rhg` helptext This will make using `rhg` more user-friendly and features more discoverable.

config: fix indentation of some`share-safe` options This makes the output much more readable.

rust-status: query fs traversal metadata lazily Currently, any time the status algorithm needs to read a directory from the filesystem (because the stat-only optimization is not available), it also stats each directory entry eagerly. Stat'ing the entries is only needed in a few cases (like when checking the mtime of a directory for caching): this patch creates a wrapper struct `DirEntry` that only stats the directory entry it represents when needed. Excerpt of an `strace` before this change on Mozilla Central: ``` openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3 newfstatat(3, "", {st_mode=S_IFDIR|0755, st_size=3540, ...}, AT_EMPTY_PATH) = 0 getdents64(3, 0x55dc970bd440 /* 139 entries */, 32768) = 5072 statx(3, ".hg", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=772, ...}) = 0 [... 135 other successful `statx` calls] getdents64(3, 0x55dc970bd440 /* 0 entries */, 32768) = 0 close(3) = 0 ``` After this change: ``` openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3 newfstatat(3, "", {st_mode=S_IFDIR|0755, st_size=3540, ...}, AT_EMPTY_PATH) = 0 getdents64(3, 0x561567c10190 /* 139 entries */, 32768) = 5072 getdents64(3, 0x561567c10190 /* 0 entries */, 32768) = 0 close(3) = 0 ```

rust-status: make `DirEntry` attributes clearer

perf-unbundle: do a quick and dirty fix to make it run on more commit Without this change, the perf commands fails within the f67741e8264b::18415fc918a1 range (boundary excluded). Check inline comment for details. With this fix, the command is able to run on this range, with a slightly different behavior (as no revset is "uninlined"). However this is still much better than not being able to run anything in this range. Especially because that range do see some performance regression for unbundle.

perf-unbundle: pre-indent the main block in per::unbundle This makes the next changeset clearer.

shelve: handle empty parents and nodestoremove in shelvedstate (issue6748)

rhg: fallback when encountering ellipsis revisions Ellipsis revisions are still experimental and buggy in non-trivial histories. We currently have no plans to improve this situation nor to add support for ellipsis revisions in `rhg`. Falling back should be done carefully (since we may have already done some work that is visible to the user), but in this case it's highly unlikely that we're doing anything useful with a repo with ellipsis revisions, so this should be strictly better, also since the error message is more explicit.

dirstate-v2: hash the source of the ignore patterns as well Fixes the test introduced in the last changeset. This caused the hash to change, which means that the check in the test had to be adapted. Since this hash is only done as a caching mechanism, invalidation does not pose any backwards compatibility issues.

dirstate-v2: add test that shows a collision in ignore patterns hash This hash is used for optimizing dirstate `status`. We demonstrate that the hash is incorrectly ignoring the changes to the semantics of the ignore files just because the contents (but not their source) haven't changed. This is fixed in the next changeset.

testlib: add `--raw-sha1` option to `f` This will be used in the patch fixing the ignore hash collision in dirstate-v2 to check the behavior of the new hashing function.

rust-status: save new dircache even if just invalidated There was a functional race between invalidating the cache (not acted upon until the end of the status algorithm) and populating the new cache (which relies upon an up-to-date version of the cache). We simply inform the cache populating function that we've just invalidated the cache for this particular directory since the information is present in the same scope.

rust-status: fix typos and add docstrings to dircache related fields

rhg: show a bug where repeated [hg status] is needed to cache everything

upgrade: no longer keep all revlogs in memory at any point Keeping all object open is unsustainable, so we will open them on demand. This mean opening them multiple times, but this is a lesser evil. Each revlog consume a small amount of memory (index content, associated nodemap, etc). While there are few "big" revlog, the sheer amount of small filelog can become a significant issue memory wise, consuming multiple GB of memory. If you combines this extra usage with the use of multiprocessing, this usage can quickly get out of control. This can effectively block the upgrade of larger repository. This changeset fixes this issue.

tests: remove non-python3 line matching and tests block We don't support Python2 anymore

demandimport: convert ignored modules from bytes -> str in extensions The default list of ignored modules are str, and test for bypassing the lazy import is `module.__name__ in ignores`, so these were effectively NOT ignored. Most of these date back to the grand byteification in 687b865b95ad, with some subsequent additions that followed the existing example. I have no idea if these modules in fact need to bypass lazy importing, but at least it follows the intent of the code.

dirstate-v2: fix edge case where entries aren't sorted See previous commit for more details.

dirstate-v2: highlight a bug when Python-packed but used in `rhg` The Python packer creates unsorted entries in the edge case that a file starts with the same name as a sibling folder. This bug has no effect on the Python `hg status` since Python ignores directories. `rhg` assumes that all on-disk entries are sorted (which is a property of the format) including folder, hence the issue highlighted. This is also technically broken in Rust-augmented `hg status`, but it makes setting up the test more complex than necessary, since it requires the packing to be Python only (which it isn't if you have Rust extensions). Fix is in the next commit.

dirstate-v2: correct documented return values of `pack_dirstate`

dirstate-v2: fix typos in docstrings

dirstate-v2: update constant that wasn't kept in sync Despite the best efforts of the comment, this constant wasn't kept in sync when the flags were being rewritten. The fact that this doesn't actually break anything in the Rust implementation too much (which does use directories) relies on the fact that all nodes can have children and that dirstate traversal is not based on that flag, but for metadata in optimizations. However the bug could become more serious should we start encoding stronger guarantees using a combination of flags including this one.

lfs: avoid closing connections when the worker doesn't fork Probably not much more than an minor optimization, but could be useful in the case of `hg verify` where missing blobs are fetched one at a time.

lfs: fix blob corruption when tranferring with workers on posix The problem seems to be that the connection used to request the location of the blobs is sitting in the connection pool, and then when workers are forked, they all see and attempt to use the same connection. This garbles everything. I have no clue how this ever worked reliably (but it seems to, even on Linux, with SCM Manager 1.58). See previous discussion when worker support was added[1]. It shouldn't be a problem on Windows, since the workers are just threads in the same process, and can see which connections are marked available and which are in use. (The fact that `mercurial.keepalive.ConnectionManager.set_ready()` doesn't acquire a lock does give me some pause though.) [1] https://phab.mercurial-scm.org/D1568#31621

keepalive: add `__repr__()` to the HTTPConnection class to ease debugging By default, this just printed the class name and memory address. By displaying the address and ports on both sides of the socket, it makes it easier to figure out what's in the ConnectionManager, and correlate with WireShark traces. It looks like the two connections mentioned in the previous commit come about because the LFS POST request to access the blobs opens connection 1, and gets a 401. Then for some reason, the follow up with credentials opens a new socket, instead of using the existing one in the pool. I have no clue why. This can be seen with something like this in the blobstore: ``` for h in self.urlopener.handlers: if hasattr(h, "close_all"): print('open connections on %s in pid %d' % (type(h), os.getpid())) for host, conns in h._cm.get_all().items(): for c in conns: print('connection: %r' % c) ```

keepalive: ensure `close_all()` actually closes all cached connections While debugging why LFS blob downloads are getting corrupted with workers, I noticed that prior to spinning up the workers, the ConnectionManager has 2 connections to the server and calling `KeepAliveHandler.close_all()` left one behind. The reason is the value component of `self._cm.get_all().items()` is a list, and `self._cm.remove()` modifies said list while the caller is iterating over it. Now `get_all()` is a deep copy of both the dict and lists in all cases.

localrepo: byteify the requirements.DIRSTATE_TRACKED_HINT_Vx warning message Flagged by PyCharm.

rhg: fallback to slow path on invalid patterns in hgignore

rhg: add a test involving hgignore lookaround

pywatchman: remove obsolete comments about importing from future See 6000f5b25c9b.

demandimport: remove an obsolete comment about importing from future See 6000f5b25c9b.

mr-template: wrap the instructions inside a comment block At least in preview mode, this hides the text so the user doesn't have to delete it. It's still visible in edit mode, so the user sees it.

revlog: use the user facing filename as the display_id for filelogs I had trouble isolating some LFS blob corruption detected by `hg verify` because the traceback referenced a file, but with the `data/` prefix in the `.hg/store` path, so it couldn't be located with the `file()` revset: ``` Traceback (most recent call last): File "/mnt/d/mercurial/mercurial/revlog.py", line 3209, in verifyintegrity _verify_revision(self, skipflags, state, node) File "/mnt/d/mercurial/hgext/lfs/wrapper.py", line 246, in _verify_revision orig(rl, skipflags, state, node) File "/mnt/d/mercurial/mercurial/revlog.py", line 158, in _verify_revision rl.revision(node) File "/mnt/d/mercurial/mercurial/revlog.py", line 1816, in revision return self._revisiondata(nodeorrev, _df) File "/mnt/d/mercurial/mercurial/revlog.py", line 1870, in _revisiondata self.checkhash(text, node, rev=rev) File "/mnt/d/mercurial/mercurial/revlog.py", line 1996, in checkhash % (self.display_id, pycompat.bytestr(revornode)) mercurial.error.RevlogError: integrity check failed on data/EXE/PPC/shrinksrec.exe:0 ``` (I'm a little surprised it resulted in a stacktrace instead of just a message, but that's a different issue. I'm also not sure how to trigger the simplestore case, since IIUC, it's also a revlog based store.) It's not clear how to handle the changelog and manifest (because the user doesn't interact with them as a file), so those cases are left alone. The other thing that would be nice to improve somehow is to indicate that the ":0" is a revlog revision, not the changeset revision that users are used to. I'm not sure how to handle the "or node" part though.

revlog: drop an unused variable assignment It's assigned again 2 lines later.

lfs: improve an exception message for blob corruption detected on transfer The message about the server crash originated in 0ee0a3f6a990 (after support for serving blobs was added), but was copied from the Facebook repo that forked prior to server side support. Therefore, this message only displayed in their client, so it was safe to assume the server crashed. But that was never the case for vanilla Mercurial, as I saw this in a server log. Also, display the blob reference so that it's easier to figure out where the problem was when a bunch of blobs are transferred at once.

branching: merge stable into default

relnotes: add 6.3

Added signature for changeset a3356ab610fc

Added tag 6.3rc0 for changeset a3356ab610fc

branching: merge default into stable This marks the feature freeze for the 6.3 release

lfs: fix interpolation of int and %s in an exception case Seen in the wild in a server log when MS antivirus was quarantining a file on the client side.

tests: catch "Can't assign requested address" in test-https.t (issue6726)

tests: add another variation of EADDRNOTAVAIL message (e.g. from NetBSD)

configitems: change the `verify.skipflags` default value to avoid a py3 crash The revlog and LFS modules use various `&` and `&=` operations with this value, which no longer treats `None` as 0. Since nothing cares if it was actually set in the config or not, just default to 0 for simplicity.

shelve: re-wrap now that the line fits

shelve: avoid setting overloading tmpwctx

dirstate-v2: skip evaluation of hgignore regex on cached directories By making the computation of [has_ignored_ancestor] lazy we're eliding its computation in the common case when none of its descendants have changed on disk. On a ~400k files repo, with a cached status, we saw a ~64% reduction in CPU time, resulting in a speedup of ~10-15% (on ZFS), and a speedup of ~38% of XFS (XFS has faster stat operations for some reason).

releasenotes: use re.MULTILINE mode when checking admonitions Release note admonitions must start at the beginning of a line within the changeset description: .. admonitions:: The checkadmonitions function search for and validates admonitions. Unfortunately, since the ctx.description is multi-line, the regex search always fails unless the admonition is on the first line. This changeset adds re.MULTILINE to the re.compile to make the re opbject multi-line.

windows: gracefully handle when the username cannot be determined This assumes implementation details, but I don't see any other way than to check the environment variables ourselves (which would miss out on any future enhancements that Python may make). This was originally reported as https://foss.heptapod.net/mercurial/tortoisehg/thg/-/issues/5835.

rhg: parallellize computation of [unsure_is_modified] [unsure_is_modified] is called for every file for which we can't determine its status based on its size and mtime alone. In particular, this happens if the mtime of the file changes without its contents changing. Parallellizing this improves performance significantly when we have many of these files. Here's an example run (on a repo with ~400k files after dropping FS caches) ``` before: real 0m53.901s user 0m27.806s sys 0m31.325s after: real 0m32.017s user 0m34.277s sys 1m26.250s ``` Another example run (a different FS): ``` before: real 3m28.479s user 0m31.800s sys 0m25.324s after: real 0m29.751s user 0m41.814s sys 1m15.387s ```

rhg: enable in case ui.statuscopies=True rhg already has code to support ui.statuscopies, but it's disabled, for seemingly no good reason.

rhg: share some code

rhg: support tweakdefaults

rhg: centralize PlainInfo

rhg: central treatment of PLAIN and PLAINEXCEPT

revset: handle wdir() in `sort(..., -topo)` The last apparent usage of `repo.changelog.parentrevs` in revsets is in `children()`, but since the sets being operated on never include wdir(), it's never called with `wdirrev` and the wdir() arg on the command line is effectively ignored instead of aborting there. I'm not sure how to fix that. Before (on a clone of hg): $ python3.8 hg perf::revset --config extensions.perf=contrib/perf.py 'sort(all(), -topo)' ! wall 0.123663 comb 0.130000 user 0.130000 sys 0.000000 (best of 76) After: $ python3.8 hg perf::revset --config extensions.perf=contrib/perf.py 'sort(all(), -topo)' ! wall 0.123838 comb 0.130000 user 0.130000 sys 0.000000 (best of 75)

revset: handle wdir() in `roots()` This is already handled in `heads()`, and both are needed to determine if a set is contiguous. I'm guessing the `0 <= p` check was to try to filter out the null revision, but it looks like that comes through in the corner case of a new repo with no commits. But that was already the case, as shown by the tests. Before (on a clone of hg): $ python3.8 hg perf::revset --config extensions.perf=contrib/perf.py 'roots(all())' ! wall 0.059301 comb 0.040000 user 0.040000 sys 0.000000 (best of 100) After: $ python3.8 hg perf::revset --config extensions.perf=contrib/perf.py 'roots(all())' ! wall 0.059387 comb 0.060000 user 0.060000 sys 0.000000 (best of 100)

pull_logger: add basic log file rotation based on size

contrib: add pull_logger extension This extension logs the pull parameters, i.e. the remote and common heads, when pulling from the local repository. The collected data should give an idea of the state of a pair of repositories and allow replaying past synchronisations between them. This is particularly useful for working on data exchange, bundling and caching-related optimisations.

shelve: do not add the dirstate backup to the transaction Otherwise the transaction will properly clean up its mess on abort… deleting the backup in the process. This break with dirstate-v2 that has more file than just the dirstate. The dirstate itself is full of various exception and is "fine" when using dirstate-v1.

branching: merge stable into default

Added signature for changeset dbdee8ac3e3f

Added tag 6.2.3 for changeset dbdee8ac3e3f

mergetools: don't let meld open all changed files on startup In meld 3.16, a multi-file change with option '-a' results in an overview list being opened. In meld 3.20, a multi-file change with option '-a' results in an overview list AND every changed file being opened. Simply removing '-a' seems to work fine for both cases and also behaves the same as before for single-file changes. As per the release notes, this is due to the following change in meld 3.19.0: "Make the --auto-compare command line flag work again (Kai Willadsen)" See also https://gitlab.gnome.org/GNOME/meld/-/issues/516

heptapod-ci: use shell script in pytype step

tests: migrate the pytype test to a shell script for easier CI processing There have been recent hangs and timeout, but it's hard to debug because the *.t test redirects output to a file and only prints it if `pytype` actually exits. This shell script can be run directly by CI, and will allow more flexibility to try to cache and restore type stubs for further speed increases.

rhg: fix bugs around [use-dirstate-tracked-hint] and repo auto-upgrade This makes two changes: - make rhg support the [dirstate-tracked-key-v1] requirement. I believe rhg never changes the tracked file set, so it's OK that it doesn't have any logic for writing this file. - fix the name of [format.use-dirstate-v2.automatic-upgrade-of-mismatching-repositories] config option in rhg, which makes rhg actually honor the auto-upgrade. These two issues cancelled each other out in tests (auto-upgrade was happening because [dirstate-tracked-key-v1] forced the fallback, not because of the config), which is I think why they went unnoticed earlier.

tests: fix http-bad-server expected errors for python 3.10 (issue6643) The format of the error message changed with this version of Python. This also removes obsolete Python 3 checks.

status: let `--no-copies` override `ui.statuscopies`

run-tests: display the time it took to install Mercurial It will help make people aware of this critical step and to assess the time it takes in various options (like a CI run for example).

run-tests: deal with distutil deprecation PEP 632 recommend the use of `packaging.version` to replace the deprecated `distutil.version`. So lets do it.

fsmonitor: migrate Python ABCs from collections to collections.abc The Collections Abstract Base Classes in the collections module are deprecated since Python 3.3 in favor of collections.abc, and removed in Python 3.10.

templates: add filter to reverse list The filter supports only lists because for lists, it’s straightforward to implement. Reversing text doesn’t seem very useful and is hard to implement. Reversing the bytes would break multi-bytes encodings. Reversing the code points would break characters consisting of multiple code points. Reversing graphemes is non-trivial without using a library not included in the standard library.

requires: re-use vfs.tryread for simplicity Avoids calling `set` twice or having to re-raise an exception and implements the routine with a single return expression.

tests: fix the flaky test test-logtoprocess.t The main change is that we're waiting for the [touched] file to appear for 5 seconds instead of 0.1 seconds. Also, instead of implementing wait-on-file from scratch, we use the existing one from testlib/ that works well.

bisect: avoid copying ancestor list for non-merge commits During a bisection, hg needs to compute a list of all ancestors for every candidate commit. This is accomplished via a bottom-up traversal of the set of candidates, during which each revision's ancestor list is populated using the ancestor list of its parent(s). Previously, this involved copying the entire list, which could be very long in if the bisection range was large. To help improve this, we can observe that each candidate commit is visited exactly once, at which point its ancestor list is copied into its children's lists and then dropped. In the case of non-merge commits, a commit's ancestor list consists exactly of its parent's list plus itself. This means that we can trivially reuse the parent's existing list for one of its non-merge children, which avoids copying entirely if that commit is the parent's only child. This makes bisections over linear ranges of commits much faster. During some informal testing in the large publicly-available `mozilla-central` repository, this noticeably sped up bisections over large ranges of history: Setup: $ cd mozilla-central $ hg bisect --reset $ hg bisect --good 0 $ hg log -r tip -T '{rev}\n' 628417 Test: $ time hg bisect --bad tip --noupdate Before: real 3m35.927s user 3m35.553s sys 0m0.319s After: real 1m41.142s user 1m40.810s sys 0m0.285s

packaging: update dulwich to drop the certifi dependency on Windows The presence of `certifi` causes the system certificate store to be ignored, which was reported as a bug against TortoiseHg[1]. It was only pulled in on Windows because of `dulwich`, which was copied from the old TortoiseHg install scripts, in order to support `hg-git`. This version of `dulwich` raises the minimum `urllib3` to a version (1.25) that does certificate verification by default, without the help of `certifi`[2]. We already bundle a newer version of `urllib3`. Note that `certifi` can still be imported from the user site directory, if installed there. But the installer no longer disables the system certificates by default. [1] https://foss.heptapod.net/mercurial/tortoisehg/thg/-/issues/5825 [2] https://github.com/jelmer/dulwich/issues/1025

tests: fix tar invocation, to address issue 6740

rhg-status: add support for narrow clones

rust: add support for hints in error messages This will be used by the narrow support code in the next commit.

rust: add Debug constraint to Matcher trait This makes sure we can easily debug which Matcher we're looking at when using trait objects, and is just generally useful. Effort to make the debugging output nicer has been kept to a minimum, please feel free to improve.

rhg: add sparse support

rhg: add debugrhgsparse command to help figure out bugs in rhg

rhg-status: extract a function for printing pattern file warnings This will be reused for the warnings produced by the sparse file parsing functions.

rust-filepatterns: allow overriding default syntax This will be used when parsing pattern files other than .hgignore like the sparse spec.

rhg: don't fallback if `strip` or `rebase` are activated Neither of these extensions do anything other than add commands, so ignoring them opens up more of the test suite to rhg.

rhg: fallback in `debugdata` if repo has `narrow` Narrow uses ellipsis nodes and debugdata does not understand them yet.

rust-status: expose DifferenceMatcher from Rust to Python

rust-matchers: implement DifferenceMatcher This can be generated by the sparse matcher.

rhg: support "!" syntax for disabling extensions This makes it so that calls in test-log.t do not fall back immediately because of the disabled extension, instead going through the CLI parsing code, which breaks because of invalid UTF-8 in a flag. I *think* clap 3.x+ supports this? I'm not sure, and we have to upgrade the minimum Rust version to use clap 3.x anyway which is out of scope for this series, so let's just kick that can down the road a little bit.

revlog: finer computation of "issnapshot" If the parent had an empty diff, we skip of it to compute a diff against the parent base. This create shorter and simpler chain. However these case could be wrongly detected as snapshot. So we improve the code doing this detection. In practice nobody care as when tried on a copy of mozilla-try and we got the same number of snapshot (1315) in both case. Performance where equivalent.

relnotes: add 6.2.1 and 6.2.2

perf: make perf::bundle compatible down to 5.2 A another small change to make it compatible with a wider set of revision. I did not check compatibility in the python-2 territory yet.

perf: make perf::bundle compatible before 61ba04693d65 A small change to make it compatible with a wider set of revision.

branching: merge stable into default

Added signature for changeset b5c8524827d2

Added tag 6.2.2 for changeset b5c8524827d2

dirstate-v2: no longer register the data-file during transaction If the data file change during the transaction, we cannot truncate it. The content of the file itself is fine as it will get backed up at the same time as the docket. Leaving the trailing data at the end of failed transaction is fine. The dirstate-v2 format supports it. The dead data will simply we written over if necessary.

fsmonitor: use new dirstate APIs (issue6728) On top of fixing fsmonitor, it moves one more "old API" use to the new one. This needs very verbose code to save a few function calls that are very expensive in Python.

dirstate-v2: backup the data file during the transaction (issue6730) If we backup of the docket, without doing a backup of the data file, we highly risk restoring a docket pointing to a missing file in the future. So we now backup the data-file alongside the docket.

dirstate-v2: display a possible issue with transaction If we backup the dirstate without the data file, we can end up in an inconsistent state of the transaction is rolled back.

test-dirstate: do not get out of the test directory It looks like we were doing one `cd ..` too many.

rust-matchers: fix behavior of `IncludeMatcher` with multiple includes This change adds a test for this fix as well as an additional test case that was useful in debugging this behavior.

compare-disco: support for `file` nodes specification This leverage the `nodefromfile` feature in core. This make it possible for callers to no longer pay the subset computation cost (and to make sure the subset is the right one, even when the base repository is different)

compare-disco: move case parsing into its own function This is open the way to the next changeset that will allow to specify a list of heads.

compare-disco: add an option to skip the case If we already know the context, we can save a lot of display space by skipping the case. This also open the way to speedup to the way we specify the subsets. (the code is hacky, but this is a quicky and dirty debug script)

compare-disco: display a header by default This help us to understand the output. (the code is hacky, but this is a quicky and dirty debug script)

compare-disco: prepare for primitive argument parsing We need to be able to configure a couple of things, so lets prepare the code for it. (the code is hacky, but this is a quicky and dirty debug script)

tests: improve portability by no longer using the flag [tar --force-local] A different way of making sure the path is not interpreted as a URL is to never give this path to [tar], instead making tar read from stdin.

phase-shelve: correct unicode string to honor 'shelve.store=internal' In the case of strip-based shelves, there should be no hidden commit found. That's because shelve.store=internal is necessary but not sufficient to enable phase-based shelves; internal-phase must also be set.

status: include `repo` in template context also for resolved paths The `repo` object needs to be in the templater context when using e.g. `relpath`. It has been missing there since it was the unresolved files were added to the templated output in 07ebb567e8bb.

shelve: remove strip and rely on prior state (issue6735)

shelve: in test for trailing whitespace, strip commit (issue6735)

shelve: demonstrate that the state is different across platforms (issue6735)

phase: rename the requirement for internal-phase (BC) The previous requirements covers both `internal` and `archived` phase. However, the `archived` phase is not ready for usage (while the internal one is mostly ready for years). So we split the archived on in a dedicated requirements (see previous changeset for details) and rename the one for internal-phase. This will avoid older client trying to use the archived phase on `internal` only repositories. Since the requirements stayed experimental since its introduction. It seems fine to drop the previous version.

phase: introduce a dedicated requirement for the `archived` phase See inline documentation for details.

phase: introduce a dedicated function to check for the archived phase The internal-phase is "ready to use" since its introduce. However, some question remains around the `archived` phase. So it seem safer to move them to separated configuration and requirements. This changeset is a first of a small series doing this.

tests: remove flakiness in a time-dependent test

rebase: add boolean config item rebase.store-source This allows to use rebase without recording a rebase_source extra field. This is useful for example to build a mirror converted from another SCM (such as svn) by converting only new revisions, and then incrementally add them to the destination by pulling from the newly converted (unrelated) repo and rebasing the new revisions onto the last old already stored changeset. Without this patch the rebased changesets would always receive some rebase_source that would depend on the particular history of the conversion process, instead of only depending on the original source revisions. This is used to implement a hg mirror repo of SvarDOS (a partially nonfree but completely redistributable DOS distribution) in the scripts at https://hg.pushbx.org/ecm/svardos.scr/ In particular, cre.sh creates an svn mirror, upd.sh recreates an entire hg repo from the svn mirror (which takes too long to do in a regular job), and akt.sh uses hg convert with the config item convert.svn.startrev to incrementally convert only the two most recent revisions already found in the mirror destination plus any possible new revisions. If any are found, the temporary repo's changesets are pulled into the destination (as changesets from an unrelated repository). Then the changesets corresponding to the new revisions are rebased onto the prior final changeset. (Finally, the two remaining duplicates of the prior head and its parent are stripped from the destination repository.) Without this patch, the particular rebase_source extra field would depend on the order and times at which akt.sh was used, instead of only depending on the source repository. In other words, whatever sequence of upd.sh and akt.sh is used at whatever times, it is desired that the final output repositories always match each other exactly.

sort-revset: introduce a `random` variant This new `sort` variant allows to shuffle any revset. It also allow for randomly picking element using `first`.

tests: remove flakiness in test-nointerrupt.t The problem was that the reaction to the signal was racing against the completion of the command. Since reaction to the signal is to print a line of warning, we can fix this by waiting for that warning to appear before allowing the command to complete.

perf: properly process formatter option in perf::unbundle Otherwise, the options are not understood.

perf: quiet stdout output in perf::unbundle There a lot of repetitive bundle application message we do not care about.

perf: quiet stderr output in perf::unbundle There a lot of repetitive transaction message we do not care about.

bisect: avoid adding irrelevant revisions to bisect state When adding new revisions to the bisect state, it only makes sense to add information about revisions that are under consideration (i.e., those that are topologically between the known good and bad revisions). However, if the user passes in a revset (e.g., '!merge()' to exclude merge commits), hg will resolve the revset first and add all matching revisions to the bisect state (which in this case would likely be the majority of revisions in the repo). To avoid this, revisions should only be added to the bisect state if they are between the good and bad revisions (and therefore relevant to the bisection). -- Here are the results of some performance tests using the `mozilla-central` repo (since it is one of the largest freely-available hg repositories in the wild). These tests compare the performance of a locally-built `hg` before and after application of this series. Note that `--noupdate` is passed to avoid including update time (which should not vary across cases). Setup (run between each test): $ hg bisect --reset $ hg bisect --noupdate --bad 56c3ad4bde5c70714b784ccf15d099e0df0f5bde $ hg bisect --noupdate --good 57426696adaf08298af3027fa77486fee0633b13 Test using a revset that returns a very large number of revisions: $ time hg bisect --noupdate --skip '!merge()' > /dev/null Before: real 0m9.398s user 0m9.233s sys 0m0.120s After: real 0m1.513s user 0m1.425s sys 0m0.052s Test using a revset that is expensive to compute: $ time hg bisect --noupdate --skip 'desc("Bug")' > /dev/null Before: real 0m49.853s user 0m49.580s sys 0m0.243s After: real 0m4.120s user 0m4.036s sys 0m0.048s

bisect: limit ancestors to revs topologically between good and bad revs Previously, when constructing its dict of revisions to their ancestors, bisect would populate the dict with ALL of the descendents of the good set, which is a bit silly because it is impossible for a revision that is a descendent of the minimum known bad revision to be the first bad rev. Instead it makes more sense to limit the revisions to just those topologically between the good and bad.

bisect: bypass changectx when translating revs to nodes When resolving the revset given by the user into node hashes, use the changelog to perform the translation rather than the repo object. This avoids the overhead of constructing a changectx which is immediately discarded.

rhg: make [rhg status -v] work when it needs no extra output Add support for verbose [status] when no extra output is actually needed. This makes it so that [rhg status] is actually useful when [tweakdefaults] is true. (since tweakdefaults implies verbose status)

perf: introduce a benchmark for delta-find That part is responsible of serious slowdown in some `hg pull/unbundle` case. So lets add a way to benchmark it.

debug-delta-find: introduce a quiet mode In quiet mode, we only print the summary of the search and skip the individual steps.

automation: set PATH when building on Windows Sometime in the 6.2 release cycle the Windows building automation broke. Building the wheel and even PyOxidizer based installers now fails with: ``` Exception: PowerShell execution failed: error: subprocess-exited-with-error Getting requirements to build wheel did not run successfully. exit code: 1 [1 lines of output] Unable to find a working hg binary to extract the version from the repository tags [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ``` I have a hunch this is a regression from upgrading pip in 1c00777702da, but I haven't verified this. It may not be, as PyOxidizer has its own bundled Python/pip. So maybe it is something in `setup.py`.

contrib: update Mercurial install in bootstrap environment

automation: transition to Windows Server 2022 Let's keep our Windows build environment modern by upgrading to the latest OS. As part of the upgrade, we pick up a migration to EC2Launch Version 2. This has a different config mechanism. So we need to port how we manage the administrator password. As part of migrating to the new YAML/JSON config file mechanism, we move the code to the powershell script that is run when the instance first launches. This ensures that the config is retained during the reboot we perform as part of building the Windows AMI. The motivation for this is I'm currently unable to build the Windows 2019 AMI due to an issue installing OpenSSH. This _just works_ on Windows Server 2022. I have no clue what the root cause is. I think it might have something to do with Microsoft not publishing the files in the right location. Differential Revision: https://phab.mercurial-scm.org/D12630

automation: refresh requirements I'm hitting errors installing the old version of cffi due to an apparent issue where older versions of cffi aren't compatible with the modern Clang I'm using. So let's upgrade packages to unbreak things and to keep things modern. Differential Revision: https://phab.mercurial-scm.org/D12629

phase-shelve: also capture the state of shelve prior to unshelve

phase-shelve: Add test for shelve technique config

phase-shelve: Implement a 'shelve.store' experimental config Accepts "internal" or "strip", indicating how the shelved changes are stored. Defaults to "internal", retaining compatibility for repos with "internal-phase" already enabled.

phase-shelve: Extract function for _target_phase

phase-shelve: expand the tests to capture use-cases supported

phase-shelve: honor and prefer obs shelves for existence and modified time

phase-shelve: read patch details from a (possibly internal) node in the repo

phase-shelve: Extract function for _optimized_match for re-use

obsstore: break the repo → obstore → repo loop This should help the garbage collector to do its job. On repository with many markers, the memory pressure from the obsstore can get quite serious.

relnotes: mention chg behavior change when given --cwd

dispatch: change cwd when loading local config Previously, the `_getlocal` function would not correctly load the repo config when given a relative `rpath` and an alternate cwd via the `wd` parameter. Normally when `--cwd` is specified, hg changes to the given directory before attempting to load the local config (and therefore does not specify a `wd`). The only time the function is called with `wd` set is when hg is running as a command server (e.g., with chg), in which case each forked worker process will attempt to configure itself via `_getlocal` before responding to the client. When given a relative repo path, the worker fails to load the repo config, detects a config mismatch with the client, and enters a redirect/respawn loop. To fix this, we can simply change to the desired working directory during config loading. (Note that simply concatenating `wd` and `rpath` won't work in all cases. The repo path could be something more complicated than a simple relative path, such as a `union:` repo.)

rust-status: ignored directories are now correctly only listed if opted into This fixes the behavior of `hg purge` removing empty ignored directory even without `--all` or `--ignored`.

test: show how purge removes ignored directories Apparently the rust code path removes ignored directories even though the pure code path doesn't. Show this in tests.

contrib: add support for rhel9

packagelib: use python3 by default

rust: bump to memmap2 0.5.3, micro-timer 0.4.0, and crossbeam-channel 0.5.0 The merge in 12adf8c695ed had conflicts in rust/Cargo.lock and rust/hg-core/Cargo.toml . Let's ignore rust/Cargo.lock - it is regenerated. For rust/hg-core/Cargo.toml, stable had dd6b67d5c256 "rust: fix unsound `OwningDirstateMap`" which introduced ouroboros (and dropped stable_deref_trait). Default had ec8d9b5a5e7c "rust-hg-core: upgrade dependencies" which had a lot of churn bumping minimum versions - also patch versions. It is indeed a good idea to bump to *allow* use of latest package. That means that major versions should be bumped for packages after 1.0, and for packages below 1.0 minor versions should be bumped too. But it doesn't work to try enforce a policy of using latest patch by bumping versions at arbitrary times. For good or bad, the merge doesn't seem to have resolved the conflicts correctly, and many of the minor "upgrade dependencies" were lost again. Unfortunately, it also lost the bump of memmap2 to 0.5.3, which is needed for Fedora packaging where 0.4 isn't available. Same with micro-timer bump to 0.4 (which already is used in rhg). crossbeam-channel bump was also lost. This change fixes that regression by redoing these "important" lines of the merge "correctly". I propose this for stable, even though dependency changes on stable branches are annoying.

tests: work around libmagic bug in svn subrepo tests libmagic 5.40 introduced a bug [1] wherein ASCII text files with fewer than 3 distinct character values would be reported as binary data rather than as text. This bug was later fixed in version 5.41 [2]. SVN uses libmagic to determine the MIME type of added files with missing or unknown extensions [3]. This results in test failures on systems with libmagic 5.40 installed: $ echo a > a $ svn add a - A a + A (bin) a A simple workaround is to change the test file's content to include 3 distinct ASCII values (including the terminating newline). [1] https://bugs.astron.com/view.php?id=180 [2] https://bugs.astron.com/view.php?id=261 [3] https://svnbook.red-bean.com/en/1.8/svn.advanced.props.html#idm2649

revlog: make _partialmatch fail fast on almost-hex inputs Before this change, resolving a revision like [0123456789^] on a large repo can take multiple seconds because: - hg does not realize this is a revset, so it tries various things, including _partialmatch(b"0123456789^") - after the rust lookup fails, it falls back to pure hg - pure hg takes all-but-last chars and converts them to binary, which *succeeds*, so it does the expensive part.

perf-unbundle: add a perf command to time the unbundle operation Check documentation for details.

perf-bundle: accept --type argument

perf-bundle: accept --rev arguments This is fairly standard nowaday.

perf-bundle: add a new command to benchmark bundle creation time

bundle: introduce a --exact option I have been wanting this options for a long time.

bundlespec: add documentation about existing option We have some documentation, lets make it complete.

Added signature for changeset f69bffd00abe

Added tag 6.2.1 for changeset f69bffd00abe

debug-discovery: apply spelling fixes from Raphaël

tree-discovery: fix the request debug output and progress location This is now associated with each request.

debug-discovery: deal with case where common is empty This code was previously confused by case where: `heads_common == {nullid}`

debug-discovery: do not abort on unrelated repositories This is a useful case to consider, so we should not abort in this case. A warning is still issued.

debug-discovery: gather the right number of roundtrips for tree discovery We where not counting the right amount of request before.

debug-discovery: also gather details on tree-discovery queries type This is useful to understand the algorithm.

debug-discovery: properly apply remote filtering in "old" mode Before this change using `--remote-as-revs` with `--old` had no effect and everything was considered as "common", which is really not what we intended.

debug-discovery: fix a typo in the doc The second option should be `--remote-…` as we just talked about `--local-…` already.

packaging: bump dulwich to 0.20.45 I'm told the new dulwich avoids hg-git test failures.

packaging: update keyring on Windows to avoid spurious stacktraces When challenged for a network password, this would spew on Windows before it actually used the stored password: ``` Error initializing plugin EntryPoint(name='libsecret', value='keyring.backends.libsecret', group='keyring.backends'). Traceback (most recent call last): File "keyring.backend", line 198, in _load_plugins init_func = ep.load() File "importlib.metadata", line 77, in load module = import_module(match.group('module')) File "importlib", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked ModuleNotFoundError: No module named 'keyring.backends.libsecret' Error initializing plugin EntryPoint(name='macOS', value='keyring.backends.macOS', group='keyring.backends'). Traceback (most recent call last): File "keyring.backend", line 198, in _load_plugins init_func = ep.load() File "importlib.metadata", line 77, in load module = import_module(match.group('module')) File "importlib", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked ModuleNotFoundError: No module named 'keyring.backends.macOS' ``` We're kinda threading a needle here because the next version of `keyring` (currently at 23.7.0) requires `importlib-metadata` 3.6+, which PyOxidizer 0.22 doesn't support[1]. [1] https://github.com/indygreg/PyOxidizer/issues/609

setup: use the full executable manifest from `python.exe` The manifest embedded by the build process (before the string here is added) already accounts for the `<requestedExecutionLevel level="asInvoker" ...>` setting. (Note that the PyOxidizer build is missing this, so it will likely trigger the UAC escalation prompt on each run.) However, using `mt.exe` to merge the fragment with what is already in the manifest seems to strip all whitespace, making it unreadable. Since Mercurial can be run via `python.exe`, it makes sense that we would have the same manifest settings (like the supported OS list), though I'm unaware of any functionality this enables. It also has the nice effect of making the content readable from a resource editor. The manifest comes from python 3.9.12. Note that this seems to strip the `<?xml ... ?>` declaration when viewed with ResourceHacker 5.1.7, but this was also the state of things with the previous commit, and `mt.exe "-inputresource:hg.exe;#1" -out:extracted` does contain the declaration and the BOM in both cases. No idea why this differs from other executables.

setup: unconditionally enable the `long-paths-support` option on Windows I don't see anything talking about why this was experimental in the first place, but maybe it was concern about the level of python2 support for it. But now, both `python.exe` and the PyOxidizer build of `hg.exe` have a manifest that enables it, so leaving it off would mean some Mercurial installations could operate on a repo with long paths, and others couldn't. Note that only the wide character functions (XxxW) will have the length restriction lifted. Sadly, distutils applies `/MANIFEST:EMBED` to the linker in a way that can't easily be turned off, so we can't use `/MANIFESTFILE` with `extra_preargs` on `link_executable`. Fortunately, the compiler object provides a path to the `mt.exe` it found during initialization, because the previous incarnation seems to have assumed it is being run within an activated Visual Studio environment. That causes MSYS builds to fail, and probably would have broke the CI environment.

setup: stop shadowing the builtin `dir` symbol I hit this when debugging what's available on the compiler.