test: explicitly "add" file before some commit in test-rollback.t `hg commit -A` will revert the `hg addremove` step if the commit fails. However `hg rollback` currently does not. We are about to improve internal consistency around transaction and dirstate and the behavior of `hg rollback` will align on the other behavior in the process. Before doing so, we make sure the test is using a separate call to `hg add` to avoid the test scenario to be affected by that future change. note: the behavior change for `hg rollback` seems fine as it affect a niche usecase and `hg rollback` usage have been strongly discouraged for a while.

rhg-files: add support for narrow when specifying a revision This makes it so that `rhg files -r NODE` works properly when using narrow.

rust-narrow: enable narrow support for plain `rhg files` Support for `rhg files -r NODE` in a future changeset.

rhg-files: make signature of `display_files` more flexible This allows the callers to use any error type that converts to `CommandError` instead of a particular concrete type.

rhg: fix user-facing error message so it matches Python implementation

rust-ui: refactor ui code for printing narrow/sparse warnings This will be used elsewhere in the code, starting from the next commit.

rhg-files: reuse centralized dirstate logic The `files` logic predates the centralized dirstate logic. It was duplicated, an didn't receive bugfixes along the way.

rust: move `filter_map_results` to public util This is a useful general-purpose function. It will be used in the next changesets.

rhg: fix typo

run-tests: stop ignoring venv-installed packages This will allow test dependencies to be installed within a venv, instead of tampering with sys/user sites. One thing to note here is that the `VIRTUAL_ENV` path takes precedence over system-site, unlike when run directly with an activated `venv`. For example, `sys.path` as viewed from a feature test in `hghave.py`, when running `hghave vcr` directly with an activated `venv`: sys.path: [ '/home/jenkins/hg-committed/tests', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/jenkins/test_venv/lib/python3.8/site-packages' ] vs `sys.path` from the same feature test, when run by `run-tests.py` with this change: sys.path: [ '/home/jenkins/hg-committed/tests', '/home/jenkins/hg-committed', '/home/jenkins/hg-committed/tests', '/home/jenkins/test_venv/lib/python3.8/site-packages', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/jenkins/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages' ]

fix: add more information to the debug output The previous debug output did not help to understand which file where fixed in which revision. The new output should cover that.

bundlerepo: raise `error.LookupError` instead of `LookupError` I'm pretty sure that this was the intention, given `error.LookupError` is raised a few lines above with the same arguments. Note that PyCharm complains about `self.display_id` (and the use above this too) being `() -> Any` instead of `bytes`, but it may be confused about the `util.propertycache` annotation on it.

unionrepo: resync several methods to actually override superclass methods Not sure how/when these skewed, but these were flagged by PyCharm as having different signatures.

hgweb: unbyteify the 100-continue check The environment seems to be `str` or `tuple[int]`, and the same check in `wsgicgi.py` is already `str` based, so I suspect this was wrong. I think what happened here is that the string wasn't `r''` prefixed in 482d6f6dba91 like `wsgicgi.py`, so it got wrongly byteified in 687b865b95ad when converting to byte literals en masse.

branching: merge with stable

setup: treat error output and non-zero return code differently Choking on Mercurial's stderr mean armless warning can derails the whole install process. We do like to be able to issue warning to people so we stop considering any stderr is a fatal mistake. This allow us to actually boostrap a version of Mercurial without loosing to many sanity point. For example in case where evolve is not loaded, something that happens when the recorded version is malformatted. Creating a wonderful ouroboros.

setup: make the version computation process more resistant The tag fetching might return and empty string¹, from there everything derails quickly. As setup tools becomes more picky about version format we make the whole seems a bit more robust. The resulting version will be obviously weird, but at least it will actually install itself. [1] This is a problem we will address in the next changesets.

bundlerepo: enforce the requirements declared by the underlying repository Previously, `hg log -r 'bundle()' -R bundle.hg` was failing for me when run from source, complaining about an unknown parent, when the system installed `hg` didn't. Some debugging showed the index was 0 length. It turned out that I didn't have the C extensions compiled, which a simple `hg log -r .` was able to indicate. The problem being that the RequirementError got handled by RepoError, which uses an empty directory as a fallback to process the bundle.

scmutil: make checknewlabel() allow "_" in otherwise numeric names (issue6737)

storageutil: match node length with repository

rust-narrow: fix loop that never loops This was caught by `clippy`. I guess the narrow tests leave something to be desired, since this previously only checked the first valid pattern.

dirstate: enforce holding the lock while doing any changes This seems like a sensible sanity check. This already caught the issue in largefile. This will catch more issue when we start using similar context manager for operation other than the parent changes.

largefile: make sure we hold the lock when updating the second dirstate The largefile extension uses a second dirstate file (and object) to track some states. In some situations, it is lazily updated when needed. These operations might not have the lock taken. This means they might conflict and race with other ongoing operations. So we now take the lock to do these operations. This was caught by the next commit.

dirstate: rename `@requires_no_parents_change` too This match the rename of the context manager.

dirstate: rename `@requires_parents_change` to `@requires_changing_parents` This match the rename of the context manager.

dirstate: write dirstate on successful exit of changing_parents context This is the first step toward having more sensible and predicatable write patterns for the dirstate. Having better write/rollback patterns will greatly reduce and clarify the needs to backup the dirstate.

dirstate: rename parentchange to changing_parents Since the new argument breaks the API anyway, we can rename it to a better name. The previous name `parentchange` might be seen as something active, a function that would directly change the parents, however this is just a context manager to frame the operation that will change the parents and adjust the dirstate content accordingly. In addition, the future sister method that will be about changes to tracking and files would have a hard time fitting in the same naming scheme in a clear way. The new naming uses a clear prefix will make it more distinct from other dirstate methods and easier to extend with other similar contexts.

dirstate: pass the repo to the `changeparent` method If we want the context to be responsible for writing (and we want it), we need to have access to a localrepository object. So we now requires a localrepository object as an argument to this context manager.

dirstate: warn about non-explicitly rolledback parent-change Now that the invalidation is dealt with by the context manager itself, we should no longer need this in the unlocking code. We start with issuing a warning if this is the case before actually dropping this code later.

dirstate: invalidate changes when parent-change fails When an error occurs during changing parents, we should invalidate all dirstate modifications and reload the dirstate. This is currently done by a `unlock` callback on the `wlock`. To fix this anomaly, we start dealing with the error directly in the context manager and its potential nesting. The "hard" part is to make sure that, when the parent-change context are nested, we and higher level nesting do not continue to use the invalidated dirstate. We introduce dedicated code to enforce that.

setup: Ensure target directory exists with building rust extension When the rust extension is the first to be build, the target directory may not exist so the copy fails.

filemerge: add union-other-first as internal merge tool See inline documentation for details.

rust: use `peek_mut` from the standard lib now that it's stable Just a little cleanup of a TODO found along the way.

clippy: actually fail if there are any warnings So far, only errors would trip the CI, this makes it so all warnings are elevated to errors, making it a CI fail if any warnings are present. This was the intended behavior, I just missed this when adding clippy.

rust-clippy: fix warning about nested ifs

hghave: refactor checks for pygments versions using checkvers()

hghave: make different has_pyoxidizer functions have different names There are 3 of them, and them all having the same name triggered code checkers.

tests: pygments 2.14+ highlight whitespace in python code

tests: make sure pygments can detect python script without extension This .t file was failing for me when running run-tests.py with python3.11. Then I tried to run it with python3.10 and it failed anyway, even though it's the default python3 interpreter. But with `python3 ./run-tests.py` it worked fine. And this is what I found while looking at the way pygments lexer checks if a file without extension is likely to be a python script: shebang_matches(text, r'pythonw?(3(\.\d)?)?') Take guess why it doesn't work for python >= 3.10. To work around this issue, we can simply hardcode an "easier" shebang for pygments. This path to python interpreter obviously doesn't need to be accurate, since we're not running this script.

convert: brz 3.3.0 moved NoSuchFile exception to breezy.transport

revset: the `random` sort should not depend on sys.maxsize (issue6770)

contrib: adjust heptapod CI flavor sizes Our CI currently runs on a number of different runners, some of which are run by Clever Cloud. By setting this environement variable, we're asking for a smaller size runner for our smaller jobs, which puts less load on the Clever Cloud infrastructure, while not affecting our overall pipeline times.

debugshell: allow commands to be specified as a CLI argument Add a `--command` option to `hg debugshell` that allows the user to pass in Python code to evaluate directly from the command line. This was inspired by the `--command` option present in Facebook's Sapling fork of Mercurial, which in turn was inspired by the `-c` option of the Python interpreter itself. It is particularly useful for writing tests, especially for getting visibility into things that otherwise aren't exposed via debug commands.

resourceutil: start using importlib.resources.files() when possible This avoids DeprecationWarnings related to our use of resources.open_binary() on Python 3.11.

heptapod-ci: add `clippy` to the CI This linter mostly makes our code more idiomatic, less surprising, has good suggestions and catches bugs. It's widely used in the Rust community and now part of the default toolchain when using `rustup`.

makefile: add `cargo clippy` to tests if cargo is available This linter mostly makes our code more idiomatic, less surprising, has good suggestions and catches bugs. It's widely used in the Rust community and now part of the default toolchain when using `rustup`.

rust-tests: run `cargo` tests inside the `rust` folder There is no reason to only run inside the `rust/hg-cpython` folder, even if the `cargo test --all` behavior ends up being the same.

tests: fix a typo in test-demandimport.t I guess we figured out why it wasn't lazy.

tests: os module is frozen in Python 3.11 (issue6786)

rust-clippy: merge "revlog" module definition and struct implementation Module inception is confusing when looking for code and almost never the right choice.

rust-clippy: merge "config" module definition and struct implementation Module inception is confusing when looking for code and almost never the right choice.

rust-clippy: upgrade `bytes-cast` dependency I released a new version of `bytes-cast` to get rid of the clippy warning, and bump to edition 2021, so let's use it.

rust-clippy: remove redundant suffix from enum Same as last time, this makes the code clearer in this instance.

rust-clippy: fix remaining warnings in `hg-cpython`

rust-clippy: disable some lints crate-wide for `hg-cpython` `rust-cpython` creates some pretty funky code that also needs to be compatible with pretty old Rust. This makes clippy quite useless in `hg-cpython` unless you disable the lints that are always triggered by `py_class!` and related. Maybe `clippy` will allow one day to exclude a dependency from its linting, but this seems quite unlikely, so this is the best we've got at the moment.

rust-clippy: fix remaining warnings in `rhg` All of these are simple changes that for the most part are clear improvements and the rest are at most equivalent.

rust-clippy: fix most warnings in `hg-core` All of these are simple changes that for the most part are clear improvements and the rest are at most equivalent. The remaining warnings have to be fixed either with a bigger refactor like for the nested "revlog" module, or in the dependency `bytes-cast`, which we own. This will be done sometime in the future.

rust-utils: strip redundant prefix from enum This was highlighted by `clippy`, I think this makes the code cleaner.

rust-filepatterns: don't `Box` subincludes unnecessarily This was caught by `clippy`.

rust-nodemap: implement `PartialEq` without allocation This was caught by `clippy`. It's probable that this ends up optimized out by the compiler, but let's not rely on that.

rust-clippy: tell `clippy` we don't need to declare a default here This is a struct only useful for tests.

rust-clippy: refactor complex type This makes reading the signature a bit less tiring.

rust-clippy: simplify return type of debug function This makes the type a little bit more readable.

rust-clippy: add `is_empty` method to please the `clippy` gods I can see it being useful anyway.

rust: don't use a reference to a `Cow` This was caught by `clippy`.

rust-clippy: reassure `clippy` that 8 arguments is expected This API is a bit verbose, but refactoring it into a struct isn't better in this particular case IMO, and there is a lot of info to pass.

rust-clippy: use `write_all` since we're not expecting a partial write Clippy caught this. This is a good lint in general, but here this was not really a bug in this example code.

rust-clippy: tell clippy we want to keep those clauses separate This makes the cases more explicit, which is wanted in this specific instance.

rust-clippy: tell clippy we care about keeping those `if` clauses separate Thses were written this way because it spells out the logic in a more explicit manner.

rust-clippy: factor an `if` with the same duplicated clause Unlike the next changeset, this if is not really made clearer by keeping the cases separate.

rust: fix broken debug assertions These were caught by `clippy`. It appears no one uses the debug builds, which is how this wasn't caught before.

rust-narrow: fix loop that never loops This was caught by `clippy`. I guess the narrow tests leave something to be desired, since this previously only checked the first valid pattern.

rust: run `cargo clippy` These automatic fixes are good to have because they make the code more idiomatic and less surprising. The transform from `sort` -> `sort_unstable` is questionable, but this is only in a test, so it doesn't matter in our case.

rust: use `logging_timer` instead of `micro_timer` I am the author of `micro_timer`. I built it at the time because I couldn't find a crate that was simple to use and flexible to do function timing with. Turns out I just couldn't find it because crates.io's search isn't all that great, or maybe I didn't look hard enough. `logging_timer` is better in every way: - supports changing the logging level - supports start and end logging - supports intermediary messages - supports inline macros - supports formatting the output - better IDE/tree-sitter integration thanks to a more robust proc macro I also changed all uses to one-liners, so it's easier to copy-paste.

pathauditor: make _checkfs_exists a static method This fixes the bug detected by pytype where the auditor used in vfs.py may be a no-op auditor (vfs.py, line 398), which doesn't have the _checkfs_exists method.

merge: add mergeresult.mapaction to improve speed As a part of [hg update] we convert all [ACTION_CREATED] merge results into [ACTION_GET] actions, and that's slightly inefficient because every insertion pays the full cost of maintaining the [mergeresult] data structure up to date. This commit adds a function [mapaction], which is faster. (saves around 0.3s on a large update involving ~400k files)

merge: avoid dereferencing repo fields repeatedly

merge: skip syntactic path checks in [_checkunknownfile] We don't need to check the paths syntactically, since they are coming from diffing the revisions, so hopefully already checked on the way in. We still need to check what's on the filesystem, to avoid traversing the symlinks or subdirs, which we can't know about statically. Also, we use the directory audit to elide [isfileorlink], this removing ~all lstat calls from hg updates from-empty.

pathutil: use `finddirs_rev_noroot` instead of `parts` The benefit this brings is very tiny, if it's even there, since we still didn't get rid of the [parts] computation. It probably won't be worth it without the subsequent patch that adds one more use of [finddirs_rev_noroot]

pathutil: add the more efficient finddir iterator (to be used in subsequent commits)

typing: import unconditionally This is needed if we're writing python3 signatures.

tests: check how hgweb handles HEAD requests This test file is loosely based on test-hgweb.t. HEAD support originally implemented in fda5a4b853ab.

convert: replace repr() by pycompat.byterepr() in cvsps.py (issue6789)

convert: change socket mode from b'r+' to 'rwb' in cvs.py (issue6789) 'r+' mode used to open sockets for read/write operations, but '+' is not supported in Python 3. We're using bytes with these sockets everywhere, so the mode should have 'b'. But the mode argument has to be str, not bytes.

convert: turn the last str regex into bytes in cvs.py (issue6789) Since root is bytes, the regular expression should also be bytes.

tests: test hg status --all with hgext/git There's a TODO item in git/dirstate.py about obtaining clean files in a more straightforward way, let's first test that status can and does show clean files at all.

tests: aborting on unknown revision emits exit code 10 even with hgext/git

worker: avoid reading 1 byte at a time from the OS pipe Apparently `pickle.load` does a lot of small reads, many of them literally 1-byte, so it benefits greatly from buffering. This change enables the buffering, at the cost of more complicated interaction with the `selector` API. On one repository with ~400k files this reduces the time by about ~30s, from ~60 to ~30s. The difference is so large because the actual updating work is parallellized, while these small reads are bottlenecking the central hg process.

diffutil: rewrite an ersatz ternary operator for building diffopts.text Pytype tends to get confused by these, but the real problem here is that if the caller passes `opts={}` as TortoiseHg has been doing, that is set as the value for `diffopts.text` when in reality it expects a boolean. When `None` is passed explicitly, the default value in `mdiff.defaultopts` is assigned.

mdiff: add a __str__ method to diffopts This makes it easier to debug by just formatting the object into `%s` to see the members and state, instead of the class and memory address.

debugshell: allow TortoiseHg builds to exit with the usual `quit()` command I've long been annoyed that `quit()` only randomly worked to exit the interpreter. When that happens, Ctrl+C doesn't work either (it simply prints "KeyboardInterrupt"), so then you have to `import sys` and `sys.exit()`. But it turns out that the behavior isn't random and it depended on which `hg.exe` was picked up on PATH first, because py2exe disables site initialization. I wasn't able to persuade the maintainer to allow an opt-in to initialization[1], but this works around it so that the behavior is now consistent however `hg.exe` is built. TortoiseHg 6.3.3 will be the first build that includes the site package, so handle the ImportError. [1] https://github.com/py2exe/py2exe/issues/154

doc: add a few comments

rhg: implement checkexec to support weird filesystems In particular, some of our repos are stored on a fileserver that simulates POSIX permissions poorly, in such a way that prevents the removal of execute permission. This causes rhg show a spurious unclean status, even though python hg reports the repo as clean. We fix this by making rhg implement the ~same checkexec logic that python hg does.

convert: stop passing str to the dateutil API in darcs I'm sure there's a bunch more stuff in here that's broken, but this was flagged by pytype.

sparse: fix a py2 based usage of `map()` In a local pytype run, this fixes: File "/mnt/c/Users/Matt/hg/hgext/sparse.py", line 386, in debugsparse: unsupported operand type(s) for item retrieval: 'fcounts: Iterator[int]' and '0: int' [unsupported-operands] No attribute '__getitem__' on 'fcounts: Iterator[int]' File "/mnt/c/Users/Matt/hg/hgext/sparse.py", line 387, in debugsparse: unsupported operand type(s) for item retrieval: 'fcounts: Iterator[int]' and '1: int' [unsupported-operands] No attribute '__getitem__' on 'fcounts: Iterator[int]' File "/mnt/c/Users/Matt/hg/hgext/sparse.py", line 388, in debugsparse: unsupported operand type(s) for item retrieval: 'fcounts: Iterator[int]' and '2: int' [unsupported-operands] No attribute '__getitem__' on 'fcounts: Iterator[int]'

histedit: byteify the help for the multifold action While there's some allowance for str in `_()`, it's commented to be for "goofy unicode docstrings in test", so no idea how well that works, but it should at least come back as bytes. With HGPLAIN, however, the str isn't touched and is returned as-is, so this seems like a real bug.

typing: use python3-style type annotation

merge: cache the fs checks made during [_checkunknownfiles] this ~halves the number of lstat calls made when updating from rev(-1) to a revision with lots of files

merge: disable the whole filesystem access loop if [_realfs] is false This makes it clearer that [auditeddir] is only relevant for [_realfs] checkers, and makes the non-realfs checkers more performant.

merge: short-circuit the _checkfs loop upon getting ENOENT This reduces the number of [lstat] calls when updating from rev(-1) to a rev with lots of files by a factor of several: for path foo/bar/baz/quux.txt without this patch we're lstatting: foo foo/bar foo/bar/baz foo/bar/baz/quux.txt and with this patch: foo foo/bar/baz/quux.txt

merge: don't pay for pathconflicts if there are none

pathauditor: no need to normcase the paths The only thing normed paths are used is the key of the caching sets, so the only change of behavior will be that the checks will be repeated for paths that differ by case. If anything, it seems correct for the check to be repeated, in case that actually affects semantics, but the main reasoning is simplifying the code and making it a bit faster. It looks like the code originally comes from commit [081e795c60e0]: it looks like that commit tried to get rid of the existing norming, but presumably did this overly cautiously, preserving it for the cache keys, even though it was pointless even then.

pathutil: slightly faster path audit in the common case

debug: add a config to abort update early This is useful to benchmark the parts of [hg update] that come before the parallel worker.

dirstate: add missing methods and kwargs to idirstate interface

dirstate: swap pathto() and get_entry() in idirstate This way the order of methods in dirstate and idirstate classes is the same. Just to make it easier to use diff tools to compare the two classes.

dirstate: update docstrings in idirstate from the current dirstate

setup: drop legacy osx compiler tuning to enable universal builds This was triggering deprecation warnings about migrating to `packaging.version` from `distutils` Version classes with `make local`. But rather than migrate that code, let's just get rid of some ~10-12 year old workarounds. As a bonus, the cext libraries that are built are now universal binaries containing x86_64 and arm64 images (at least when built on macOS 11.4 with Xcode 12.5 and the universal version of Python 3.9.13). Several things to note here: - Apple dropped support for 10.15 in Nov 2022, and OS X Lion that is referenced is 10.7 (unsupported since late 2014) - `xcode4` was basically always True because of the `>=` check (10.8 used Xcode 5, and I have Xcode 10.2 on 10.14) - `xcode51` was always False for modern-ish Xcode, because of the exact version string matching - Python 3.8 only supports OS X 10.9+; the Python 3.9.1+ universal installer is macOS 11+ only, and Python 3.10 drops the x86_64 installer to deliver only the universal installer. All of this is to say, the only thing lost by dropping this code on modern Xcode is that `os.environ['ARCHFLAGS'] = ''` is no longer set. But we probably shouldn't be setting that anymore, as shown by the universal libraries now being generated. I was able to `make local` and `python3 run-tests.py --local` with python 3.9.9, Xcode 10.2, and macOS 10.14.6, and didn't incur any more than the usual few test errors, so this should still work on some older versions of macOS.

tests: simplify `(py3 no-py36 !)` output matching predicates to `(no-py36 !)` It's all py3 now.

tests: drop `(py3 !)` output matching predicates Presumably these were paired with `(no-py3 !)` at one point, but now they were unconditionally true. test-check-code.t required a couple of `(glob)` markers on the changed lines in test-lfs-serve-access.t, because of the `$LOCALIP` usage on those lines. Not sure how those lines slipped through the checks previously.

run-tests: drop a workaround for python2 The problem is only on python3, but the awkward handling was because python2 didn't have this exception type. I've sporadically seen it running in WSL, but no clue what it means.

contrib: drop py2 support from testparseutil.py

tests: drop py2 support from `f` utility

tests: drop py2 support from test-doctest.py

tests: drop py2 support from test-demandimport.py

tests: optional PEP 657 error location in test-lfs-serve-access.t (issue6782)

tests: optional PEP 657 error location in test-extension.t (issue6781)

tests: filter out PEP 657 error locations in tracebacks (issue6780) https://docs.python.org/3/whatsnew/3.11.html#whatsnew311-pep657

pytype: add coverage for parts of hgext There are tons of things to fix here (which have been blacklisted for now), but this should help prevent further regressions.

convert: stop passing str to the dateutil API in darcs I'm sure there's a bunch more stuff in here that's broken, but this was flagged by pytype.