rust-index: add `pack_header` support

rust-index: support cache clearing I'm not 100% sure how useful it is outside of perf, but it's still worth implementing.

rust-index: check rindex and cindex return the same get_rev This is a temporary safeguard while we synchronize both indexes.

rust-index: synchronize remove to Rust index Future steps will bring the two indexes further together until we can rip the C index entirely when running Rust code.

rust-index: remove `__setitem__` method from the mixed index This is not defined on the Python or C one, and isn't used anywhere.

rust-index: check equality between rust and cindex for `__len__`

rust-index: synchronize append method We now append to the Rust index just as we do to the C index. Future steps will bring the two indexes further together until we can rip the C index entirely when running Rust code.

rust-revlog: teach the revlog opening code to read the repo options This will become necessary as we start writing revlog data from Rust.

rust-index: pass data down to the Rust index This will allow us to start keeping the Rust index synchronized with the cindex as we gradually implement more and more methods in Rust. This will eventually be removed.

rust-index: add append method This is the first time the Rust index has any notion of mutability. This will be used in a future patch from Python, to start synchronizing the Rust index and the C index.

rust-index: add an abstraction to support bytes added at runtimes In order to support appending data to the Rust index, we need to abstract data access away from the immutable (on-disk) bytes, to seemlessly fetch either from the preexisting data or from the newly added data.

rust-mixed-index: move the mmap keepalive into a function The same code will be used for keeping the new index mmap around.

rust-mixed-index: rename variable to make the next change clearer We're going to add another mmap reference holder, so let's rename this one first.

rust: fix cargo doc for hg-cpython

branching: merge with default We merge with the current children of the bad merge (37b52b938579)

branching: merge with stable This recreates `37b52b938579` right as a `hg branch --rev 5b186ba40001` screwed up the content.

changelog: disallow delayed write on inline changesets Since this will never happens, we can make the situation invalid and to stop to handling the associated the case. This simplify the random access file reading too.

changelog: never inline changelog The test suite mostly use small repositories, that implies that most changelog in the tests are inlined. As a result, non-inlined changelog are quite poorly tested. Since non-inline changelog are most common case for serious repositories, this lack of testing is a significant problem that results in high profile issue like the one recently fixed by 66417f55ea33 and 849745d7da89. Inlining the changelog does not bring much to the table, the number of total file saved is negligible, and the changelog will be read by most operation anyway. So this changeset is make it so we never inline the changelog, and de-inline the one that are still inlined whenever we touch them. By doing that, we remove the "dual code path" situation for writing new entry to the changelog and move to a "single code path" situation. Having a single code path simplify the code and make sure it is covered by test (if test cover that situation obviously) This impact all tests that care about the number of file and the exchange size, but there is nothing too complicated in them just a lot of churn. The churn is made "worse" by the fact rust will use the persistent nodemap on any changelog now. Which is overall a win as it means testing the persistent nodemap more and having less special cases. In short, having inline changelog is mostly useless and an endless source of pain. We get rid of it.

test-transaction-safety: glog out irrelevant flag The test is focussing on the inline flag, so we glob out the other to highlight that fact and prevent noise in the future.

test-transaction-safety: perform the test on a filelog This test previously checked the transaction safety of splitting the changelog. The changelog is a special case, with delayed/diverted writes and we will stop inlining it soon. So we keep testing that transaction is safe around inline on another revlog type : a filelog. Minor comestic adjustement will be done in the next changesets.

test: clarify test-parseindex offsets We will make this revlog non-inline, so we clarify the code to make sure it is simple to adjust the test later.

test: use more globing for perf timing Not sure why we kept the number here.

branching: merge with stable

persistent-nodemap: avoid writing nodemap for empty revlog The format cannot encode the lack of tip_rev. There is currently nothing known to write such empty nodemap right now, but the change we are preparing on default reveal this issue. So I had rather fix it on stable.

histedit: remove superfluous echo() and endwin() calls (issue6859) ncurses patchlevel 20231111 started returning an error from endwin() if called twice without a intervening screen update. Per Sven Joachim in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1058041#17: "AFAICS, invoking curses.echo() and curses.endwin() is superfluous because curses.wrapper already does that for you, and calling curses.endwin() twice throws an error with the newer ncurses. Removing those two lines should fix the problem."

statprof: handle `lineno == None` in more cases This continues the work from 972f3e5c94b8. We saw a crash on line 956 but I updated lots of other places as well.

branching: merge stable into default

Added signature for changeset 71bd09bebbe3

Added tag 6.6.1 for changeset 71bd09bebbe3

relnotes: add 6.6.1

procutil: move stdin assignment outside of try-finally block There is an stdin variable in the global scope of this module. And in the `finally` block of this try-finally statement we're checking `if stdin is not None`. Let's make sure we don't confuse code check tools into thinking we want to use global stdin by moving this line of code outside of `try`. This was caught by pytype 2023.11.21 on Python 3.11.2.

zeroconf: give inet_aton() str instead of bytes All other uses of this function in this extension are already fixed (i.e. use strings instead of bytes). This was caught by pytype 2023.11.21 on Python 3.11.2.

revlog: avoid wrongly updating the data file location on "divert" If we are in the inline case, we need to align the location of the "data" file with the temporary location of the file (i.e. "00changelog.i.a"). However we should not do that for non-inline case… and before this changeset we had been doing it. In addition `index_file` is already a property taking care of updating the "segment file" filename when needed. So we can simply remove all that code. As a result, code trying to read the diverted data before they were committed ended deeply confused as the "00changelog.i.a" file is nothing like the "00changelog.d" file. However nothing corrupted data as all writing where properly handled outside of the "segment file". In "best" cases this small in-memory corruption of the filename when unnoticed until the transaction was committed or rolled back and in the worse case, some data reading was failing during the transaction and resulted in the transaction to be rolled back. However wrong data never reached the disk, so this bug should be have corrupted any repository. This is not catch by tests because most test use a small repository and therefor an inline revlog. In addition the bug only triggers when a changelog read is done in the following "rare" situation: - after some delayed write - after that data have been written in a "divert" file (i.e. `00.changelog.i.a`) - before transaction commit - outside of a "writing" context The issue was introduced in d83d788590a8

revlog: avoid exposing delayed index entry too widely in non-inline revlog Before this change, the index entry would be seen as "appended" to the data file. It did not hurt too much as there are never accessed for reading, but this was odd. So lets stop doing so.

revlog: add one more assert about state of thing when splitting This assert is currently happy, but it does not hurt to adds it to clarify expected state and catch potential error in the future.

add: don't attempt to add back removed files unless explicitly listed This fixes the bug demonstrated by the previous patch.

tests: show failure to `hg add -I` a dir->symlink transition

setup: try a non-pure version of the local Mercurial if the pure fails Things like `zstd` can make the pure version fails.

setup: make debug simpler by adding a `__repr__` to `hgcommand` This help when trying to debug this logic.

censor: fix things around inlining The temporary revlog cannot go through the inline → split process as this would break at transaction commit. (that might be fixable, but lets keep things simple for now). We introduce a cleaner way to enforce this as the previous one was broken in 6.6 On the way we remove multiple weird, fragile and broken overwrite of revlog attributes and we focus on passing the configuration across. We also had to update the test to actually create a non-inline revlog.

revlog: add a `may_inline` argument to revlog This allow for a clean skipping of the inline feature when needed, for example by censor.

revlog: allow explicit passing of config to revlog This will be useful to fix censor in a later changeset.

censor: show that the `not-inline` → `inline` test is broken The source revlog should not be inlined and it is…

rhg: support rhg status --rev --rev

rust: add a utility function to merge ordered fallible iterators Adding a function merge_join_results_by, a version of itertools::merge_join_by that works on "fallible" iterators (iterators that can produce errors)

rhg: refactor hg status, make the display code usable for non-dirstate status

perf-tags: fix clear_cache_fnodes to actually clear that cache The function was not doing it what it advertise for a long time. So we fix it and we add a way for the perf extensions to detect broken version.

perf-tags: fix the --clear-fnode-cache-rev code It seems like this code never run?

tests: do not fail tests in a state with uncommitted .py file removal The problem is that [hg locate] lists removed files too. We use [hg files] instead because that does not list removed files.

tests: fill in the Windows pattern for `$EADDRNOTAVAIL$` matching This fixes test-https.t on Windows. It looks like the real error translation is "Cannot assign requested address.", and the message here is the start of a longer description, so I'm not sure why this part is emitted. But it's not worth digging into, as it's evidently the same failure.

tests: avoid a cascading failure on Windows The `identify --debug` command here on Windows emits skip updating dirstate: identity mismatch because of the debug switch, which got captured and added to `.hgtags`, and then hijinx ensued. The point of `--debug` seemed to be to get the long hash, so just do that with templating. I have not idea if the message is indicating a problem- there seems to be many more of them in other tests that are not present on Linux.

phabricator: stringify the argument to `getattr()` This fixes a "TypeError: getattr(): attribute name must be string" crash and subsequent test spew. This likely broke in 18c8c18993f0 and was missed in CI because `pytest-vcr` is missing.

py3: pass unicode strings to hasattr() throughout I guess we missed these instances because they're not covered by tests. This is a graft of 1625fe807c04a490f9516bc8e14140e570c06146 that landed on default.

debugformat: speedup the "plain-cl-delta" check Checking every single revision is too slow, let's use a sampling approach.

templatekw: fix inconsistency of diffstat with diff.merge Previously, `-T'{diffstat}'` was giving stats from the diff against p1, regardless of whether `--config diff.merge=yes` is set. This was inconsistent with `log --patch` which is aware of that option.

util: move diff_parent from logcmdutil to diffutil This function will be used outside of the log command (in templatekw, used by hgweb, for which logcmdutil is not available). Let's move this function together with the rest of the diff-related utils instead.

logcmdutil: return structured diffstat data for json

cleanup: drop the `bytes` compatibility for attribute related function We can rely on the builtins directly now.

cleanup: turn `wrapfunction` deprecation warning into an error We could simply drop the check, but lets raise explicit error instead of suffering strange error in case of misuse.

cleanup: turn `wrappedfunction` deprecation warning into an error We could simply drop the check, but lets raise explicit error instead of suffering strange error in case of misuse.

cleanup: turn `pathsuboption` deprecation warning into an error We could simply drop the check, but lets raise explicit error instead of suffering strange error in case of misuse.

cleanup: drop deprecated config attribute on the revlog class This code was marked for deletion in the next cycle. We are now in the next cycle.

cleanup: drop `path.pushloc` deprecated since 6.5 This was scheduled for removal, let us remove it.

cleanup: drop `dirstate.is_changing_parent` deprecated since 6.5 This was scheduled for removal, let us remove it.

cleanup: remove some code scheduled to be removed after 5.9 looks like we missed this when cleaning up previous code.

branching: merge stable into default

relnotes: add 6.6 There are no differences between the rc and the actual release.

Added signature for changeset 26c57e7a0890

Added tag 6.6 for changeset 26c57e7a0890

py3: pass unicode strings to hasattr() throughout I guess we missed these instances because they're not covered by tests.

Added signature for changeset 27055614b685

Added tag 6.6rc0 for changeset 27055614b685

relnotes: add 6.6rc0

branching: merge default into stable for 6.6rc0

unstable: do not consider internal phases when computing unstable The revisions that are not part of the "working" set by other means should not be considered for the evolution related computation. This impact the test introduced in 5f9af8422b31 as this is actually a more semantic fix of the issue.

unstable: use the `_mutablerevs` function when computing content divergent This is a useful function to get the revision relevant to these computation, lets make sure all code use it, so that we can improve that `_mutablerevs` function in a later changeset.

unstable: use the `_mutablerevs` function when computing phase divergent This is a useful function to get the revision relevant to these computation, lets make sure all code use it, so that we can improve that `_mutablerevs` function in a later changeset.

rust: add explicit resolver field to top-level cargo manifest file Since Rust 1.51.0, Cargo has support for a better dependency resolver. Workspace packages (like ours) need to explicitely set the field, so let's. https://doc.rust-lang.org/edition-guide/rust-2021/default-cargo-resolver.html

rust: run a clippy pass with the latest stable version Our current version of clippy is older than the latest stable. The newest version has new lints that are moslty good advice, so let's apply them ahead of time. This has the added benefit of reducing the noise for developpers like myself that use clippy as an IDE helper, as well as being more prepared for a future clippy upgrade.

rust-clippy: ignore clippy's recommendation for "useless" cast See explanation inline.

branching: merge stable into default

Added signature for changeset c083d9776cb2

Added tag 6.5.3 for changeset c083d9776cb2

relnotes: add 6.5.3

revlog: avoid opening and closing the file for each cloned revision The previous code was flushing files after each new revision, slowing things down. For exemple, with this change, the evolve repository can run `hg debugupgraderepo --run --optimize re-delta-parent` in about 3.4s instead of 4.5 seconds.

censor: accept censored revision during upgrade They can simply be passed by as censored.

censor: show that censored revision prevent repository upgrade This is not great.

smartset: don't ignore hidden revs when intersecting This fixes the bug I demonstrated in the previous commit, but I'm not sure at all if it's the right way of doing it.

tests: demonstrate crash in `unstable()` with internal-phase orphans

rust-matchers: fix quadratic complexity in `FileMatcher` Concretely, this command: ``` $ echo hg up -r <nodeid>; time hg revert dir1 dir2 -r <othernode> --debug hg up -r <nodeid> real 0m14.690s user 0m14.766s sys 0m5.430s ``` was much slower despite using 16 cores before this change. The approach taken here is the same one used in match.py, in exactmatcher. This changeset was originally written by Valentin Gatien-Baron in a private repository. I have redacted the commit message and did a minor clean up of the code.

revlog: add a small cache of unfiltered chunk This can provides a massive boost to the reading of multiple revision and the computation of a valid delta chain. This greatly help operation like `hg log --patch`, delta computation (helping pull/unbundle), linkrev adjustment (helping copy tracing). A first round of benchmark for `hg log --patch --limit 1000` shows improvement in the 10-20% range on "small" repository like pypy or mercurial and large improvements (about 33%) for more complex ones like netbeans and mozilla's. These speeds up are consistent with the improvement to `hg pull` (from a server sending poor deltas) I saw benchmarking this last year. Further benchmark will be run during the freeze. I added some configuration in the experimental space to be able to further test the effect of various tuning for now. This feature should fit well in the "usage/resource profile" configuration that we should land next cycle. When it does not provides a benefit the overhead of the cache seem to be around 2%, a small price for the big improvement. In addition I believe we could shave most of this overhead with a more efficent lru implementation.

revlog: minor refactor in the chunk gather process We will introduce some caching in this method in the next changeset, we make some of the most "disruptive" change first as touching this could break (and maybe did during the development process).

changelog-delay: move the delay/divert logic inside the (inner) revlog Instead of hacking throught the vfs/opener, we implement the delay/divert logic inside the `_InnerRevlog` and `randomaccessfile` object. This will allow to an alternative implementation of the `_InnerRevlog` that does not need to use Python details. As a result, the new implementation can use the transaction less agressively and avoid some extra output since no data had been written yet. That seems like a good side effect.

revlog: add a `canonical_index_file` attribute on inner revlog This is currently the same and the index_file but it will become more complex when we start doing delayed write.

changelog-delay: move "delayed" check to a more official API To avoid reaching inside the inner object in the future, we needs some official API. We put one of such API early to reduce the size of the final diff.

changelog-delay: move the appender class next to randomaccessfile We want to move the delay/divert logic at the revlog level (to have all IO related logic in the _InnerRevlog) we start with small piece that are easy to move on their own.

changelog-delay: adds some check around delaying and diverting write Theses assert shows we never call delay or divert if the revlog hold file handle on the revlog's file.

revlog: consolidate cache invalidation within the inner objet The invalidation of data that belong to the inner object can and should be delegated to the inner object.

shelve: drop some weird manually stripping before transaction abort I cannot find a case where this is actually useful, so lets triggering strip in the middle of uncommitted transaction.

revlog: move entry writing in the inner object This is the second big piece of API we want to live on the inner object. With this the inner object have freedom it needs to implement things internally. (except for the delayed write feature of the changelog, that will be migrated in coming changesets)

revlog: move `sidedata` in the inner object Same object as rawtext, but for sidedata.

revlog: move the `rawtext` method on the inner object This is one of the current core goal of the inner object: having a "simple" API for some of the core revlog action, like getting a raw text, with the inner object able to do that part internally, on its own.

revlog: move the_revisioncache on the inner object The goal for this inner object is to compute that things we cache, it make more sense to have the inner object handle it directly.

revlog: move the `deltachain` method on the inner object This is a necessary step before being able to move more logic around restoring a revision content there. For now, we do a simple patch for the perf extension logic, when the implementation of the inner object changes, we will likely need some evolution of the API. However this is true of many things in the perf extension. So we will see this later.

revlog: move the `_chunks` method on the inner object This is a necessary step before being able to move more logic around restoring a revision content there. For now, we do a simple patch for the perf extension logic, when the implementation of the inner object changes, we will likely need some evolution of the API. However this is true of many things in the perf extension. So we will see this later.

revlog: add a couple more of useful method on the inner object This will be needed for the next changeset.

revlog: move the `_chunk` method on the inner object This is a necessary step before being able to move more logic around restoring a revision content there. For now, we do a simple patch for the perf extension logic, when the implementation of the inner object changes, we will likely need some evolution of the API. However this is true of many things in the perf extension. So we will see this later.

revlog: move the compression/decompression logic on the inner object This is a necessary step before being able to move more logic around restoring a revision content there. For now, we do a simple patch for the perf extension logic, when the implementation of the inner object changes, we will likely need some evolution of the API. However this is true of many things in the perf extension. So we will see this later.

revlog: move the splitting-inline-revlog logic inside the inner object This is another large IO block that we need to move within the inner object if we want's it to be self sufficient.

revlog: synchronise the various attribute holding the index filename The segmentfile and the attribute need to be synchronized, let's enforce that.

revlog: drop reference to docket in the inline-splitting code revlog with a docket do not use inline revlog and do not need to split them. So we can remove some code handling docket there.

revlog: move _getsegmentforrevs on the internal object See inline documentation for details.

revlog: create a iteration of a _InnerRevlog object within the revlog The goal of this object is to isolate a sub-API that can be implemented by a compiled object (e.g. Rust). So the boundary of this object will be arbitrary depending of what can we easily implemented in the Compiled code. For now, we start simple, and move the code that manage the IO objects in the inner object. More will come in the coming changesets. Note: the object definition could live in the different module to thin the `revlog.py` file, however there are other better candidate for extraction first and I have enought patch stacked on top of the this one for the split in this patch not to be worth it. So I leave this to future me.

revlog: drop the unused `_chunkcache` attribute Apparently, some time ago, the chunk cache moved in the randomaccessfile object.

revlog: drop the unused `_sidedatareadfp` method It has no caller anywhere and is probably the remains of some older code.

revlog: also migrates `revlog.upperboundcomp` to ConfigClass This was planned but overlooked when doing the rest of the migration.

revlog: small doc to the `files` method

revlog: remove the `_indexfp` method The method is only used two time and we want to move that code to a lower level object. So we simply inline the method to simplify further refactoring.

revlog: avoid opening and closing the file for each cloned revision The previous code was flushing files after each new revision, slowing things down. For exemple, with this change, the evolve repository can run `hg debugupgraderepo --run --optimize re-delta-parent` in about 3.4s instead of 4.5 seconds.

censor: accept censored revision during upgrade They can simply be passed by as censored.