stabletailgraph: clarify excl part size computation

stabletailgraph: clarify naiveness of current implementation Both the naive and the actual versions of the algorithms are going to co-exist for the tests. This makes is clearer that this one is naive.

stream-clone: introduce the notion of an experimental "v3" version We introduce a new experimental "v3" stream protocol, disabled by default. In practice the "v3-exp" protocol introduced in this changeset is identical to v2, but this changeset, lay the groundwork for having a new protocol: configuration, capability exchange, test coverage, etc. The actual protocol work will starts in the coming changesets.

stream-clone: check is a compatible protocol can be found The previous code was explicitly checking if "v2" is listed in the "stream" bundle2 capability. The new code is checking is there is anything common between the versions supported client side and server side overlaps. This prepare the introduction of more stream version than "v2".

stream-clone: bail-out earlier if stream clone is not requested The `canperformstreamclone` function is bit messy. However it seems clearer to me to check if a stream-clone have been requested by the client or the server at all, before checking if a compatible protocol can be negotiated with the server. So I am doing some gratuitous movement so reorder conditional.

stream-clone: bail-out earlier if pull is partial The `canperformstreamclone` function is bit messy. However it seems clearer to me to process the very generic condition about "can we consider a stream-clone at all", before checking if a stream-clone is requested and if a compatible protocol can be negotiated with the server. So I am doing some gratuitous movement so reorder conditional.

stream-clone: bail-out earlier if destination repo is not empty The `canperformstreamclone` function is bit messy. However it seems clearer to me to process the very generic condition about "can we consider a stream-clone at all", before checking if a stream-clone is requested and if a compatible protocol can be negotiated with the server. So I am doing some gratuitous movement so reorder conditional.

stream-clone: check the version of streaming clone supported by the client Make the server refuse to produce streaming clone bundle, if the client doesn't specify the stream=v2 capability. This is in preparation to introduce stream=v3. As far as I can tell, this capability was added at the same time as support for bundle2-based streaming pulls was added, so I don't expect clients to break. (the modern client doesn't break, at any rate)

stream-clone: make sure the `stream` capability is set when bundling This is important to start narrowing protocol option in the next changesets.

stream-clone: upgrade the error message for bad stream request The new version if more compact and more consistent with the general Mercurial usage.

stream-clone: yield cache entry in `_entries_walk` too The new function now cover the same ground as _v2_walk.

stream-clone: introduce a _entries_walk That function insert itself between the store and `_v2_walk`. It only deals with StoreEntry unlike `_v2_walk` that deal with actual file. A share of the `_v2_walk` logic will be moved in this new `_entry_walk` function. Having this function will help us to implement a new "v3" version of the protocol that will be more entry centric.

store: make `walk` return an entry for obsolescence if requested so Instead of having dedicated code in the streamclone code, we should have the store deal with advertising the data it contains.

store: yield phases before changelog Creating the `changelog.i` file make the repository usable, so dealing with phase earlier seems better.

store: make `walk` return an entry for phase if requested so Instead of having dedicated code in the streamclone code, we should have the store deal with advertising the data it contains.

cli: add a test of `hg debugnodemap --manifest` this is a new option that's not tested yet

cli: fix spelling in `debugnodemap` error messages

cli: make debugnodemap capable of inspecting an arbitrary nodemap This lets us inspect the manifest nodemap and individual directory nodemaps when treemanifest is used.

rust: mostly avoid streaming zstd decompression Streaming ZStd decompression seems slightly slower, and the API we use makes it very inconvenient to re-use the decompression context. Instead of using that, use the buffer-backed version, because we can give a reasonable-ish size estimate.

rust: in zstd decompression, avoid a useless vec initialization

rust: speed up zstd decompression by re-using the decompression context Admittedly, zstd is already pretty fast, but this change makes it a bit faster yet: it saves ~5% of time it takes to read our large repo. The actual motivating use case is treemanifest: in treemanifest we end up reading *lots* of small directories, and many of them need decompression, and there the saving for [rhg files] is >10%. (which also seems unreasonable, we should probably keep things uncompressed more)

store: rename `topfiles` to `top_entries` The method is now returning StoreEntries let us rename the method for clarity.

store: rename `datafiles` to `data_entries` The method is now returning StoreEntries let us rename the method for clarity.

store: use boolean property for upgrade's matchrevlog

store: use the new boolean property in `upgrade`

store: use the new boolean property in `remotefilelogserver`

store: use the boolean property in `repair_issue6528`

store: use the new boolean property in `narrow`

store: use the boolean property in `store`

store: introduce boolean property for revlog type This will avoid exposing implementation details to more generic code.

store: issue a single entry for each revlog We now yield a single entry, with data about each files involved. This help to simplify multiple code using this and it will help to further simplify and fixes the streaming code.

store: rename `unencoded_path` to `entry_path` for StoreEntry This remove the ambiguity with StoreFile and make sure use code will be using the right API.

store: do the revlog matching on entry directly This is the last blocker to safely merge the revlog files in a single entry.

store: split the wrapping of encodedstore between _wrap and datafiles The `datafiles` method of `basicstore` is doing a lot of work that should be done on decoded filename. So we now wrap `_walk` to do the decoding, and less work in `datafiles`. This is necessary to make sure file from the same revlog can be grouped together.

store: introduce a main_file_path method for revlog This help code that need to point revlog to an index file. This is put to use in the upgrade code.

upgrade: actually use StoreEntry API to create revlog Lets make use of the semanctic of the object we are passed.

upgrade: use StoreEntry object in upgrade We will make more use of the API in the next changeset, but just moving to use entry is a significant change for the engine codebase.

upgrade: drop a quick fix that is not longer necessary We won't issue bad revlog from topfile anymore.

store: use StoreEntry API instead of parsing filename in largefile This is more explicit and more robust.

store: use StoreEntry API instead of parsing filename when listing manifestlog This is more explicit and more robust.

store: use StoreEntry API instead of parsing filename when fixing issue6528 This is more explicit and more robust. We also introduce a small output change as it make things simpler and this is a affecting a debug-command.

store: use StoreEntry API instead of parsing filename in remotefilelog This is more explicit and more robust.

store: use StoreEntry API instead of parsing filename in narrow This is more explicit and more robust.

store: add a `target_id` attribute on RevlogStoreEntry This hold the "target" (file, directory, etc) of a revlog. Having this available will help a lot of code to avoid direct file path access.

store: actually tag tree manifest revlogs as manifest revlogs It turn out we have been mislabeling these for a long while. This is now fixed.

store: also gather files per revlog in `topfiles` This conclude out revlog gathering.

store: also group files by revlog in fncache version of datafiles One more step.

store: add logic to group revlog file together For now each file get its own entry, this will help stopping this, soon™. We use such gathering in the `basicstore` code.

store: change `_walk` return to `(filename, (type, size))` If we are to group file per revlog, having the filename as the "main key" will be useful. This change will make the following changes clearer.

store: lazily get file size on demand for the fncache case We don't have the information in the first place, so we can avoid querying the file system inconditionnaly for use case we don't needs it. This change requires the StoreFile class to be passed a vfs to retrieve the file_size if needed. In the non-fncache case, we already have the information from file system walking, so we keep it and use it.

store: only access is_volatile information through the file object This make sure other code only access this information through the proper API, and it prepare for store entries to be able to agregate multiple files.

store: only access file_size information through the file object This make sure other code only access this information through the proper API, and it prepare for store entries to be able to agregate multiple files.

store: have custom init for entries class This will get useful to add special processing later in this series.

store: use specialized class for store entries We introduce two different classes for revlog and other entries. For now, we still have multiple entry for the same revlog, but we will work toward grouping the different file in a single entry in this series. Having the distinction is a step toward this goal.

store: introduce a EntryFile object to actually access file info For now a StoreEntry match a single file, but the goal is to eventually combine multiple file in a higher level Entry, so we need to introduce this distinction and use it first.

store: use a StoreEntry object instead of tuple for store files We want to make the store return more semantic information instead of a stream of file path. To achieve this, we start with adding a simple object that hold the same information as the tuple it replace, and do a simple update to the user code to fetch and use the same information. From there, we will be able to iteratively upgrade the codebase toward better objects.

store: no longer explicitly mark the splitted index as ignored Since the fncache is going to automatically recognised it as non "normal" revlog file, we don't need to explicitly make it ignored.

store: cleanup what is recognized as a revlog file The temporary file from `censor` are not relevant to the store as they are just temporary. So let `fncache` and `datafiles` forget about them.

store: only inclure revlog related file in `datafiles` This is a "reading" equivalent of the previous patches (see that patch for details).

store: do not record file that are not part of a revlog in fncache The fncache exist to list file to copy/stream when cloning a repository, it should only contains file that are relevant for a revlog in such case. For example, temporary file are not relevant. So we now skip the addiction of non-relevant file to the fn-cache in the first place.

revlog-split: expand the some test to cover the success case too It seems useful to double check the case of a transaction successfully committed before trying the one with transaction abort.

transaction: use a ".bck" extension for all backup file There are too much code in Mercurial that assume file function from extension. Keeping the original extension when doing backup is just creating confusion (for exemple, backup of revlog's confused with new version revlogs). So we change the backup name from now on to be explicilty backup file and prevent this kind of problem. (note that repository using fncache is less affected (but still affected) by this kind of problem as the backup are not listed in the fncache).

repair-issue6528: remove a now useless continue The condition for that continue have been excplicly filtered out of the `files` list a handful of line earlier.

rebase: don't call rewriteutil.precheck() with to-be-skipped commits It's clearly incorrect to call `rewriteutil.precheck()` for commits that we're not about to rewrite. We haven't noticed yet because the function doesn't check for divergence, but I'm about to teach it to do that. Differential Revision: https://phab.mercurial-scm.org/D10259

copies: make sure deleted copy info do not overwriting unrelated ones See test documentation for details. This is yet another corner case for copy tracing. Differential Revision: https://phab.mercurial-scm.org/D9200

utils: helper function to print top memory allocation site The memorytop function uses Python's tracemalloc module to show the source lines / backtraces with the largest remaining allocations. This allows identifying the origins of active memory by placing calls in strategic locations. Allocations from C extensions will show up as long as they are using the Python allocators. Differential Revision: https://phab.mercurial-scm.org/D9236

branching: merge stable into default

stabletailgraph: clarify omission of linear parts in test sorts Those are not interesting and are collapsed using globs. Only the tagged nodes are interesting.

stabletailgraph: omit uninteresting linear parts in test log The test uses tags for the nodes of interest. There is no need to print the linear parts which exist only to force the rank.

stabletailgraph: fix typo in test

stabletailgraph: fix terminology in doc

Added signature for changeset fc445f8abcf9

Added tag 6.4.3 for changeset fc445f8abcf9

relnotes: add 6.4.3

backup: fix issue when the backup end up in a different directory Because of store encoding, we might end up with the backup in a different directory than the initial copy (for example if the backup path make it cross the 120 char limit). This can create crash, especially since 6.4 where backup are used during revlog split. Making sure the directory exists fixes these crash We added a test covering this case. Strictly speaking, this has always been broken, however the new code in 6.4 triggers it more easily.

vfsproxy: inherit the `createmode` attribute too It is an important part of the API when creating directory. We will need it in the next changeset.

revlog: test more complex file pattern for revlog split There have been a report of failure while splitting revlog. The file name involved was important. The pattern involved here are not problematic, but it help to setup the machinery to test multiple files, the actual fix and problematic file will arrive in a later changeset.

repo-upgrade: write new requirement before upgrading the dirstate This will prevent a small race condition where another hg process still believes the repo is dirstate-v1 during the upgrade process. This is good to have, but it is not a proper fix for the underlying problem. There is code that assumes a requirement means a usage, e.g. having the `generaldelta` requirement would imply *all* revlogs to use general delta, but it's not true, it simply means that the repository advertises to the client it needs to understand `generaldelta` in order to read the repo. In the case of the dirstate, having the requirement *technically* should always be the same as using dirstate-v2, since there is only one dirstate and requirements should be as minimal as possible. However, we should not assume this and make the code more robust in a future patch (series).

perf: introduce a `perf::stream-locked-section` command This command benchmark the initial part of a stream clone, where the repository is locked.

transaction: tr._names are actually bytes, use byte string to join them Looks like this is never checked in the tests? But trying e.g. `print(tr)` by hand before this change would give: TypeError: sequence item 0: expected str instance, bytes found Looks like tr._names are always bytes (although this isn't actually enforced, but maybe at some point it will at least be type checked e.g. by pytype).

nix: add nix-defined package and devel env (flake) This adds a Nix Flake which defines: - package recipies to build and run Mercurial (C, C+Rust variants), - a meta package to run the test suite in an isolated sandbox, - a development environment with some tools (notably Python and the Black formatter, pinned to match the CI). Python is pinned to the recommended version. The Rust toolchain is a fairly recent one provided by the Nixpkgs channel. It is not yet pinned to the same version as the "reference" Debian package, but this does not seem to cause any issue. Example usage of local commands are provided in the `flake.nix` file. Once merged, it should also be possible to pull and run directly from the source repository directly with something like: `nix run hg+https://foss.heptapod.net/mercurial/mercurial-devel?ref=TOPIC&dir=contrib/nix' -- version`

hgignore: ignore nix result symlink

tests: add test ignore list for nix Some tests cannot run properly in the nix sandbox. This adds a list of tests to ignore when running in that context.

ui: keep the progress bar around when writing if stdout is not a tty

rhg: correctly relativize copy source path

rhg: don't print copy source when --no-status is passed

tests: add test for status copy source formatting

fix: highlight the required configuration and behavior of the fixer tools The problem is that `hg help fix` didn't mention *how* to configure the tools, and while I knew that `{rootpath}` existed in the configuration, I missed that the tools require reading content from stdin. (I configured `gofmt` to use `{rootpath}`, and that had the effect of squashing all changes in a file at `.` into the first commit and emptying that content from its descendants.) Basically all this does is put a pointer in the default (command level) help to the extension level help that mentions the configuration, and moves the extension level help that documents reading from stdin and writing to stdout to the top to give it more prominence. The last sentence is adjusted a bit to reflect the new location.

debug-delta: add minimal documentation for `devel.bundle-delta` option So the next person won't have to grep the code to know what it is doing.

delta-find: simply code that is now never invoqued without general delta Finish early in on-general delta case mean a lot of code can assume that general delta is not enabled.

delta-find: add a simple safeguard to prevent bad non-general-delta This make it easier to catch request to build delta that does not make sense in the general delta world. It seems better to raise a programming error than to corrupt the repository.

delta-find: never do anything fancy when general delta is off Without general delta, the only option is to do a delta against the previous revision in the revlog, or to do a full snapshot. So no need to dive into the more complicated logic. This prevent Mercurial to misbehave when applying remote deltas when `pulled-delta-reuse-policy` was set to "try-base" or "forced" and would produce and store invalid deltas. We add an associated tests to make sure this scenario is covered by tests. There are more code to cleanup after this, but I what to keep the fixing patch simple.

delta-find: assume the target-rev if not specified This will be useful in the next patch, but I am putting it in its own commit to make sure we can easily bisect any subtle side effect. (none are expected, but still)

tests: in filterpyflakes, tolerate non-ascii file contents

comments: fix spelling

chg: set CHGHG before connecting to command server cf4d2f31 (!523) changed chg to set `CHGHG` itself when spawning a new command server, in order to ensure that the path to the `hg` executable would be checked during server validation. (This is useful when chg is built with `HGPATHREL`). However, that change broke chg because it failed to set `CHGHG` before trying to connect to an existing command server. This means that if `CHGHG` is not present in the environment, chg will always spawn a new command server, entirely negating the point of chg. This breakage wasn't initially caught because of the difficulty of writing automated tests with the `HGPATHREL` feature enabled, which meant the change was only tested manually to make sure that it fixed the problem with `HGPATHREL` that prompted the change. In practice, this functionality is only really useful when chg is built with `HGPATHREL`, so I considered wrapping it in an `#ifdef` to preserve the old behavior by default. However, this makes it hard to write tests since one would have to explicitly set `HGPATHREL=1` when running `run-tests.py` (which is why the original change lacked tests). It would be great if there were a way of testing features that are gated behind conditional compilation.

chg: declare environ (issue6812)

heptapod-ci: display python version it will be useful to debug in the CI

branching: merge stable into default

relnotes: add 6.4.2

Added signature for changeset f952be90b051

Added tag 6.4.2 for changeset f952be90b051

revlog-split: make sure the self._indexfile attribut is reset (issue6811) Before this change, after a transaction committing a file split, a revlog object would have its `self._indexfile` attribute desynchronised from the actual file storing the data. If that same object is reused (as we do for the manifest during clone bundles), this lead to the data being writting in the wrong location and the repository to go corrupt. We not properly reset the attribut when applicable and everything is back in working order.

revlog-split: also check a case where bundle contains multiple changegroup This case is not affected, we remains in the same transaction. However it seems important to improve the test coverage around these "features". It also helped to assert the scop of the critical bug.

revlog-split: show manifest data loss situation when using clonebundle After the commit of the transaction from applying the clone bundle, changes to the manifest from the subsequent pull would write index data in the wrong file. This lead to data being "lost" after the pull and the repository to be corrupted. We add a test checking this specific scenario. In practice, the issue only affects the manifest but we also test the issue with a file for future proofing things up. See test documentation and failure for details.

sha1dc: Make sure SHA1DC_BIGENDIAN is set on Darwin/PowerPC __BYTE_ORDER__ is not defined on old toolchains so sha1dc gets built in little endian mode on a big endian platform, hilarity ensues. Tested on Mac OS X Tiger 10.4.

hidden: add support to explicitly access hidden changesets with SSH peers This implements support for using --remote-hidden with an SSH server. The remote `hg serve --stdio` call is passed the `--hidden` flag as a request to access hidden changesets. This approach has benefits similar to the one we used for HTTP peers. It * works around the lack of global parameters in wire protocol v1, * reuses the `--hidden` flag (that does not use the wireproto), and * can be safely ignored by older client (fitting the best effort contract). Same as for HTTP, the feature is experimental so we have all the room we needs to update the implementation in the future if deemed necessary. The SSH version of the `--remote-hidden` config uses the same configuration as the HTTP support to control the access to this feature. The name of the user running the command is used for the checking. Test written by Pierre-Yves David.

hidden: support passing --hidden with `serve --stdio` This currently has no effect since the server code will filter the repository again. However, it is clearer to put this change in its own changeset. This will mostly impact ssh peers, clients being now able to pass --hidden when spawning the server. See the next changeset for details on how this is going to be used.

hgweb: move ismember from `hgweb.common` to `scmutil` To implement hidden access control for other protocol than http, we need code outside of hgweb to access this logic.

hidden: add support for --remote-hidden to HTTP peer Test written by Pierre-Yves David.

hidden: add support to explicitly access hidden changesets via HTTP We now detect hidden access request and validate them while handling HTTP wireprotocol command. This changeset only introduces server support. Client support is added in a subsequent changeset. The mechanism we use for the HTTP wireprotocol is the same as the one used to lift filtering for hgweb. The use of the global `access-hidden` parameter in the http url is a simple and effective approach that * works around the lack of global parameters in wire protocol v1, * reuses the mechanism used for hgweb (that does not use the wireproto), and * can be safely ignored by older client (fitting the best effort contract). The feature is still experimental, so we can easily update the implementation details in the future. However, landing a first implementation now will help test the feature in the wild and iterate on alternative implementations if deemed necessary.

hidden: support for explicitly accessing hidden changesets in wireproto server This installs the basic infrastructure to be able to access hidden changeset through the wireprotocol. Each wireprotocol server still needs a way to define how the feature is triggered.

hidden: add support for --remote-hidden to localpeer

pull: add --remote-hidden option and pass it through peer creation This option will allow to pull changesets that are hidden on the remote. This is useful when looking into a changeset’s evolution history, resolving evolution instability or mirroring a repository. The option is best effort and will only affect the pull when it can. The option will be ignored when it cannot be honored. Support for each type of peer is yet to be implemented. They currently all warn about lack of support. The warning code will get removed as peers gain support for this option. The option is still experimental, so we will have freedom to update the UI or implementation before it graduates out of experimental. Based on a changeset by Pierre-Yves David, which added the option.

hgweb: add support to explicitly access hidden changesets This changeset adds a "global" `access-hidden` argument to hgweb. This argument lift the "hidden" filtering. This means the request has access to hidden (eg: obsolete) changesets. Secret changesets remains filtered. This feature has multiple applications. The first main use case is to allow the hgweb interface to display more obsolescence related data, such as the Anton Shestakov work to add `obslog` support to hgweb. The second foreseen usecase is support for a `--remote-hidden` argument to `hg pull` and `hg clone`. This flag will make it possible to retrieve hidden (typically obsolete) changeset under some conditions. This is useful when digging up obsolescence history or when doing full mirroring. More on this feature coming in later changesets. To avoid exposing information by mistake, access to this feature is currently controlled with the `experimental.server.allow-hidden-access` config option. The option works the same way as `web.allow-push`. The current default is to not allow any hidden access. However we might change it before the feature stop being experimental.

clone-bundles: add an option to generate bundles in the background This is what the "on-change" version have been doing from the start, it seems useful to also offer this option in the command.

clone-bundles: add a command to clear all bundles

clone-bundles: add more test cases for automatic generation See inline documentation for details.

clone-bundles: introduce a command to refresh bundle This command allows for a different mode than the one attached to the transation. It might fit some hosting setup better.

clone-bundles: add a configuration to control auto-generation on changes We are about to introduce a "manual" way to deal with automatic clone management but running a command using some internal API. The first step is to introduce a way to control the "on changes" behavior.