dirstate: add default value to _addpath We want to remove the magic value usage from the higher level, so lets stop passing them explicitely when possible. Differential Revision: https://phab.mercurial-scm.org/D10930

dirstate: move the _rangemask filtering closer to its storage The goal is to get ready to move these kind of processing at a lower level. We start with move move _rangemask filtering close to where it is sent to the lower level to make the future move trivial. Differential Revision: https://phab.mercurial-scm.org/D10929

dirstate: introduce a symbolic constant for the AMBIGUOUS_TIME marker This is going to be clearer and easier to track than -1. Ultimately I would like to get ride of this special value everywhere but in the lower level, however we need to clarify the API first. This changeset is part of such clarification. Differential Revision: https://phab.mercurial-scm.org/D10928

dirstate: introduce a symbolic constant for the NONNORMAL marker This is going to be clearer and easier to track than -1. Ultimately I would like to get ride of this special value everywhere but in the lower level, however we need to clarify the API first. This changeset is part of such clarification. Differential Revision: https://phab.mercurial-scm.org/D10927

dirstate: introduce a symbolic constant for the FROM_P2 marker This is going to be clearer and easier to track than -2. Ultimately I would like to get ride of this special value everywhere but in the lower level, however we need to clarify the API first. This changeset is part of such clarification. Differential Revision: https://phab.mercurial-scm.org/D10926

dirstate: split a not-so-one-liner This is shorter and simpler to read. Differential Revision: https://phab.mercurial-scm.org/D10925

dirstate: split a not-so-one-liner This is simpler to read. Differential Revision: https://phab.mercurial-scm.org/D10924

dirstate: split a not-so-one-liner This is shorter and simpler to read. Differential Revision: https://phab.mercurial-scm.org/D10923

dirstate-tree: Keep a counter of descendant nodes that have an entry … and change the `DirstateMap::has_dir` method to be based on this counter being non-zero instead of the mere presence of a node. A node with zero descendent with an entry currently should be removed from the tree, but soon we’ll make the dirstate track additional nodes. (Specifically, for non-ignored directories in order to keep track of their mtime and optimize status by doing fewer `read_dir` calls.) Differential Revision: https://phab.mercurial-scm.org/D10922

dirstate: Removed unused instances of `DirsMultiset` … in Rust-backed dirstatemap. The Python class `dirstatemap` had cached properties `_dirs` and `_alldirs` that were not used for `hastrackeddir` and `hasdir` since they were redundant with corresponding fields for the Rust `DirstateMap` struct. `dirfoldmap` is modified to reuse instead the directory iterator introduced in 3b9914b28133c0918186b6e8b9e4f1916e21338d. Differential Revision: https://phab.mercurial-scm.org/D10921

dirstate-v2: Use 32-bit integers instead of 64-bit for offsets This saves 12 bytes per node. (Nodes representing files or directories.) These are offsets to other parts of the file. This would only be a limitation for a `.hg/dirstate` file larger than 4 GiB, which would only happen for a repository with dozens of millions of files and directories. Differential Revision: https://phab.mercurial-scm.org/D10920

status: Extend read_dir caching to directories with ignored files See code comments Differential Revision: https://phab.mercurial-scm.org/D10909

dirstate-v2: Drop cached read_dir results after .hgignore changes Soon we’ll want the status algorithm to be able to skip `std::fs::read_dir` in more cases, notabling when listing unknown files but not ignored files. When ignore patterns change (which we detect by their hash, added to the dirstate-v2 format in a previous changeset), a formerly-ignored file could become unknown without changing its parent directory’s modification time. Therefore we remove any directory mtime from the dirstate, effictively invalidating the existing caches. Differential Revision: https://phab.mercurial-scm.org/D10907

censor: split the core of the logic into its own function We now have a "generic" rewrite function (only able to do censoring for now) and a thin wrapper that implement the `censor` API with it. We are now ready to start incorporating strip specific changes. Differential Revision: https://phab.mercurial-scm.org/D10904

censor: migrate the logic to a set of `censor_revs` Instead of considering a special unique censored revision within the code, we consider a set of revision (currently always of size 1). This make the main code less censor-centric and prepare for the usage of a similar approach for stripping changesets. Differential Revision: https://phab.mercurial-scm.org/D10903

censor: rename `rl` to `revlog` in the main function Now that the bulk of the code moved to smaller function we are less restricted on line length and we can use more explicite naming. Differential Revision: https://phab.mercurial-scm.org/D10902

censor: extract the part about creating and opening new files in a function The v2_censor function is huge, now that its content has settled a bit it is a good time to split individual part inside dedicated function. The last part is the file copying and opening logic. It now have its own function. Differential Revision: https://phab.mercurial-scm.org/D10901

censor: extract the part about recomputing delta in a function The v2_censor function is huge, now that its content has settled a bit it is a good time to split individual part inside dedicated function. It is now the turn of the logic that recompute new delta to replace the one based on the revision that are going away. Differential Revision: https://phab.mercurial-scm.org/D10900

censor: extract the part about writing the other revision in a function The v2_censor function is huge, now that its content has settled a bit it is a good time to split individual part inside dedicated function. We continue with a small function that add a non-censored revision back to the revlog. Differential Revision: https://phab.mercurial-scm.org/D10899

censor: extract the part about writing the censored revision in a function The v2_censor function is huge, now that its content has settled a bit it is a good time to split individual part inside dedicated function. We start with a small function that process the censored revision. Differential Revision: https://phab.mercurial-scm.org/D10898

censor: put the tuple of open files in an explicit variable This will be simpler to pass these file around in future changesets. Differential Revision: https://phab.mercurial-scm.org/D10897

revlog: rewrite `censors.py` to `rewrite.py` The logic we use for censoring revision will be mostly common with the one we needs for stripping. So we rename the module to `rewrite` to better match its future content. Differential Revision: https://phab.mercurial-scm.org/D10896

Backed out changeset 9b8f326731ac Unfortunately, disabling the rust extensions means newer Mercurial versions no longer have the persistent-nodemap feature enabled. This means a Mercurial 5.8.1 on RPM-based Linux distributions will no longer be able to read repositories created by a Mercurial 5.8 on RPM-based Linux distributions. This violates the compatibility rules (see https://www.mercurial-scm.org/wiki/CompatibilityRules ). For this reason, I have to backout this change. I'll try to find another solution to the 'hg purge' crashes. Differential Revision: https://phab.mercurial-scm.org/D10918

copies: Keep changelog sidedata file open during copy tracing Instead of having a callback that opens and closes that file many times, a add and use a context manager method on the `revlog` class that keeps files open for its duration. Differential Revision: https://phab.mercurial-scm.org/D10888

purge: restore the `testedwith` declaration Otherwise, this extension gets highlighted as a 3rd party extension when an exception occurs. Differential Revision: https://phab.mercurial-scm.org/D10895

rust: remove dead utils module This was introduced a while back while not being imported anywhere in the module system. How pertinent this Facebook module was to the codebase was debatable at the time, nowadays with the Rust VFS layer having laid its foundations, it is clear that we should drop it. Differential Revision: https://phab.mercurial-scm.org/D10890

rebase: use str-keyed opts in remaining places Differential Revision: https://phab.mercurial-scm.org/D10892

rebase: keep str-keyed opts long enough to make `action` a str This is just another little py3 cleanup. Differential Revision: https://phab.mercurial-scm.org/D10891

benchmarks: restore `output` variable lost in D10884 Noticed by pyflakes (which I didn't have installed for Python 3.9 when I sent D10884). Differential Revision: https://phab.mercurial-scm.org/D10894

censor: implement censoring for revlogv2 It is a bit verbose and rough, but it works. Most of that logic can be common for `stripping`, so we can expect more refactoring of that code to accommodate both needs. However I wanted to keep this changesets "simple enough" and before moving forward. We also need to properly delete the older index/data/sidedata file, but this has implication for streaming clone and transaction, so this will come later. Differential Revision: https://phab.mercurial-scm.org/D10869

deltas: at a `target_rev` parameter to finddeltainfo Otherwise, recomputing a delta for a revision might result in a delta against a later revision or a full snapshot thinking we are appending a new revision. We will make use of this during censoring (and later, stripping). Differential Revision: https://phab.mercurial-scm.org/D10887

revlog: list older-but-still-around file in `files` These older files will eventually be removed, but we have to take them in account until they are. So we update `files` to keep fncache happy. Differential Revision: https://phab.mercurial-scm.org/D10868

revlog: add docket method to request new content files If we want to write content into new files, we need to be able to ask for them. Differential Revision: https://phab.mercurial-scm.org/D10867

revlog: add a way to keep track of older uids in the docket When the revlog content is rewritten, we will use new files, to avoid truncating the previous ones. We need some way to keep track of the older files before we clean them up. Differential Revision: https://phab.mercurial-scm.org/D10866

revlog: factor the logic to determine the delta compression out This make the logic clearly isolated and documented and it will help use to reuse it during censors/strip. Differential Revision: https://phab.mercurial-scm.org/D10889

nodemap: fix some comment formatting Black seems unhappy about it. Differential Revision: https://phab.mercurial-scm.org/D10865

revlogv2: fix `hg verify` with revlog v2 We added a bunch of new files but we never teached some internal about it. This is now fixed. Differential Revision: https://phab.mercurial-scm.org/D10858

revlogv2: also test that local clone works This is now the case so lets add a test. Differential Revision: https://phab.mercurial-scm.org/D10857

clone: also report the bookmark file as copied This is a small UI adjustement, but this is easy enough to do. Differential Revision: https://phab.mercurial-scm.org/D10856

clone: reuse the stream clone logic for local clone Streaming clone and local (non `--pull`) clone do mostly the same thing, however they were using different logic to do so. This means the logic frequently went out of sync and that new case had to be dealt with twice. This is fragile and anoying. So we replace this with a re-use of the logic we use for streaming clone. I can see various test changes: - a more precise progress output, - armless fncache loading during clone, - fncache is no longer hardlinked (since we write it by hand). I am not reinstalling the `reposimplestore` specific output, as far as I understand this variant have been broken for years. Differential Revision: https://phab.mercurial-scm.org/D10855

copyfiles: add a way to relax the file system checking for hardlink This is critical for transaction file, less for hardlink/copy clone as we are about to do. Since `pure` build does not have a `getfstype` implementation this would disable hardlink clone for all pure build. So we add a parameter to control that extra check. Differential Revision: https://phab.mercurial-scm.org/D10854

copyfile: add a option callback for failed hardlinking Local clone, adjust its UI depending on the success of using hardlinking, so we add a small callback making it possible for `copyfile` to signal if the requested hardlinking failed. Differential Revision: https://phab.mercurial-scm.org/D10853

streamingclone: extract the scanning part from the generation part We will reuse the scanning part for local clone, so we need it in a dedicated function. Differential Revision: https://phab.mercurial-scm.org/D10852

vfs: add a `register_file` method on the vfs class This is used by the fncache vfs to register new file. Until now, `fncache` have been doing this "automatically" by monitoring write pattern. However this is fragile and when we copy files in place by other means, we need something more robuts. So we add an explicit method to do so. Differential Revision: https://phab.mercurial-scm.org/D10851

clone: use "official" API to create local clone destination This make sure we have a properly created, fully functional repository early. This will be useful to simply the hardlink/copy phases of the local clone to make it share more of its logic with the similar "stream" cloning. This has a minor impact of the test and the resulting repository has is better initialized (eg: the `wcache` directory is pre-created.) Differential Revision: https://phab.mercurial-scm.org/D10850

localrepo: introduce a clone_requirements function This function take a source repository and return a relevant set of requirements that should be used by a copy clone. This will help make the creation of the destination repository during copy clone simpler. Differential Revision: https://phab.mercurial-scm.org/D10849

createrepository: allow to directly pass the target requirements This is useful when doing a local clone that copies store contents, it will requires the destination to use the very same store requirements so directly providing them will be simpler and safer Differential Revision: https://phab.mercurial-scm.org/D10848

narrowbundle: use new context manager for silencing the ui Same reasoning as the previous change. This affects a few tests because of the hack from d7304434390f (changegroup: move message about added changes to transaction summary, 2019-09-08). Differential Revision: https://phab.mercurial-scm.org/D10886

debugbackupbundle: use new context manager for silencing the ui A difference between setting `ui.quiet` and using `ui.silent()` is that the latter also silences `ui.write()` calls. That's practically always what one wants, including here, I think. Differential Revision: https://phab.mercurial-scm.org/D10885

ui: add a context manager for silencing the ui (pushbuffer+popbuffer) We often silence the ui by calling `ui.pushbuffer()` followed (a later in the code) by `ui.popbuffer()`. These places can be identified by the fact that they ignore the output returned from `ui.popbuffer()`. Let's create a context manager for these cases, to avoid repetition, and to avoid accidentally leaving the ui silent on exceptions. I deliberately called the new function `silent()` instead of `buffered()`, because it's just an implementation detail that it uses `pushbuffer()` and `popbuffer()`. We could later optimize it to not buffer the output. Differential Revision: https://phab.mercurial-scm.org/D10884

shelve: don't swallow errors that happen when reverting the working copy Same reasoning as the parent commit. The code in question here comes from fb8065de47b0 (unshelve: silence internal revert, 2014-08-15), which says that it was to prepare for some upcoming revert changes. I suppose those changes have happened by now but we fixed something about it, because I don't see any unwanted changes in tests (no changes at all in tests, in fact). Differential Revision: https://phab.mercurial-scm.org/D10883

histedit: don't swallow errors that happen when updating the working copy Commit 34165875fa5d (histedit: limit updated and merging output to important updates, 2016-02-03) added a call to `ui.pushbuffer(labeled=True, errors=True)`. I don't know if there was a reason for the `errors=True` (nor the `labeled=True` bit, which I removed in the previous commit). It seems bad to just discard the errors, so let's not do that. By not doing that, we can also remove a workaround for the lack of errors from `test-removeemptydirs.t`. Differential Revision: https://phab.mercurial-scm.org/D10882

histedit: don't record labels for output that we will discard Differential Revision: https://phab.mercurial-scm.org/D10881

dog-raft: work with str-keyed opts throughout Differential Revision: https://phab.mercurial-scm.org/D10864

cmdutil: make resolvecommitoptions() work on str-keyed opts As with `checknotesize()`, I also changed to snake_case while at it, to help extensions a little. Differential Revision: https://phab.mercurial-scm.org/D10863

cmdutil: make amend() take str-keyed opts I couldn't think of a reasonable change to the signature here, so unfortunately it keeps the existing one. Differential Revision: https://phab.mercurial-scm.org/D10862

commit: keep opts dict str-keyed a bit longer Differential Revision: https://phab.mercurial-scm.org/D10861

dog-raft: keep opts dict str-keyed a bit longer Differential Revision: https://phab.mercurial-scm.org/D10860

cmdutil: make checknotesize() work on str-keyed opts Since this patch changes the behavior of the function, I consider it a bonus if the signature also changes so that extensions can detect which version they're dealing with (for the record, I don't have any such extensions). Therefore, I also dropped an unused `ui` argument and renamed the function to use snake_case. Differential Revision: https://phab.mercurial-scm.org/D10859

amend: work mostly with str-type **opts for simplicity Differential Revision: https://phab.mercurial-scm.org/D10880

revlog: use file read caching for sidedata The previous changeset extracted this caching logic from the revlog class into a new class. Adding a new instance of that class allows using the same logic for side data. Differential Revision: https://phab.mercurial-scm.org/D10879

revlog: Extract low-level random-access file read caching logic The `revlog` class does many things, among which fulfilling requests for arbitrary byte slices from the revlog "data file" by reading a larger chunk and caching it in memory, in order to reduce the number of system calls. This extracts that logic into a new class, so that it may later also be used for the side-data file (with another instance of that class). The copyright notice of the new file does not include a date or author name since such information tend not to be kept up-to-date: https://www.linuxfoundation.org/en/blog/copyright-notices-in-open-source-software-projects/ Differential Revision: https://phab.mercurial-scm.org/D10878

typing: disable warnings for a couple of Windows specific attributes Differential Revision: https://phab.mercurial-scm.org/D10870

typing: suppress a false error in mercurial/revlogutils/docket.py on py2 `ord()` wants bytes or str on py3, so I'm guessing it got confused by passing a single byte instead of a one byte string. But this seems to work on 2.7.18 anyway. Differential Revision: https://phab.mercurial-scm.org/D10876

transaction: simplify `undo.backupfiles` file creation with a variable This make the code shorter and clearer. (This is an opportunity improvement while looking at something next to that.) Differential Revision: https://phab.mercurial-scm.org/D10846

transaction: narrow the error filtering when failing to rename undo file Having inconsistent "undo" files can be quite serious so we narrow the error filtering to the intent that the comment explain. (This is an opportunity improvement while looking at something next to that.) Differential Revision: https://phab.mercurial-scm.org/D10845

transaction: clarify a conditionnal about version check Let us move the short branch early. Differential Revision: https://phab.mercurial-scm.org/D10844

transaction: extract message about different version in a constants This make the code a bit simpler and clearer. (This is an opportunity improvement while looking at something next to that.) Differential Revision: https://phab.mercurial-scm.org/D10843

transaction: explain why some recovery failed Right now we issue a message about failing to recover some file, but not why. It seems useful to add some information about that. (This is an opportunity improvement while looking at something next to that.) Differential Revision: https://phab.mercurial-scm.org/D10842

copyfiles: deal with existing file when hardlinking If the hardlinking fails, we fallback to `shutil.copy`, but do not consider future hardlinking doomed. This is a slight improvement from the current situation, we still avoid hardliking in a case we might be able to do it. However this does not have an impact of the rest of the operation. (This is an opportunity improvement while looking at something next to that.) Differential Revision: https://phab.mercurial-scm.org/D10841

clone: cleanup the "cleanup dir" logic used during local clone With the previous code, there was a small windows were we could delete too much data. This should no longer be the case. (This is an opportunity improvement while looking at something next to that.) Differential Revision: https://phab.mercurial-scm.org/D10840

config: add an experimental option to list all known config That option is not ready for prime-time, hence the `exp-` prefix. However, this is a good base to start going toward completion. This is also quite useful for developer on its own for now. Differential Revision: https://phab.mercurial-scm.org/D10356

revlog: byteify a few error strings Differential Revision: https://phab.mercurial-scm.org/D10875

revlog: fix a typo closing the wrong file Caught by pytype as possibly being None, but the file is closed in the previous conditional. Differential Revision: https://phab.mercurial-scm.org/D10874

py3: byteify the version string passed to the deprecation warning method The other callers were already correct. Differential Revision: https://phab.mercurial-scm.org/D10873

urlutil: byteify several localized messages Flagged by pytype. Differential Revision: https://phab.mercurial-scm.org/D10872

urlutil: use the deprecation warning function on the `ui` object PyCharm flagged this as not existing, and I suspect a copy/paste error. Differential Revision: https://phab.mercurial-scm.org/D10871

dirstate-v2: Store a hash of ignore patterns (.hgignore) Later, this help extend `read_dir` caching to directories that contain ignored files (but no unknown files). Such cache must be invalidated when ignore patterns change since a formerly-ignored file might become unknown. This helps the default configuration of `hg status` where unknown files must be listed, but ignored files are not. Differential Revision: https://phab.mercurial-scm.org/D10836

rhg: propogate error coming from HgError::Abort to CommandError Differential Revision: https://phab.mercurial-scm.org/D10839

rhg: add exit code to HgError::Abort() My previous attempts to have rhg end with correct exit code was more of bug hunting. I found cases which were failing and fixed them. But as one might expect, more tests started failing. Let's add exit code `HgError::Abort()` and make it users explicitly tell what exit code they want. Differential Revision: https://phab.mercurial-scm.org/D10838

rhg: split non_repo_config and `--config` loading in different functions This will help us in better handling of error caused when trying to load `--config` values. Differential Revision: https://phab.mercurial-scm.org/D10837

rhg: look for repository in ancestors also instead of cwd only Last patch introduced config reading at startup to parse value of `--repository` flag. However, that patch only tried to check for current repository at current working directory and not it's ancestors. This patch fixes that. Differential Revision: https://phab.mercurial-scm.org/D10767

rhg: read [paths] for `--repository` value hg parses `-R` and `--repository` CLI arguments "early" in order to know which local repository to load config from. (Config can then affect whether or how to fall back.) The value of of those arguments can be not only a filesystem path, but also an alias configured in the `[paths]` section. This part was missing in rhg and this patch implements that. The current patch still lacks functionality to read config of current repository if we are not at root of repo. That will be fixed in upcoming patches. A new crate `home` is added to get path of home directory. Differential Revision: https://phab.mercurial-scm.org/D10296

merge: with stable

revlog: allow to pass an existing docket to `_loadindex()` This will be when switching undelying files, for examples during censors and strip operation with rvlog-v2. Differential Revision: https://phab.mercurial-scm.org/D10800

revlog: add a ways to blacklist some revision when searching for a delta This will be useful to recompute appropriate deltas one the fly during censor/strip operation with revlog-v2. Differential Revision: https://phab.mercurial-scm.org/D10799

util: add `nb_bytes` argument to `copyfile` to partially copy a file When set, this allow to copy only the first `nb_bytes` of a file. This will be useful for censor/strip operation with revlogv2. Differential Revision: https://phab.mercurial-scm.org/D10798

revlog: move `revisioninfo` in `revlogutils` We will need it in other utility module. So lets extract it from `revlog.py`, the module is too large already anyway. Differential Revision: https://phab.mercurial-scm.org/D10797

revlog: use the `entry` function in bundlerepo We can use some of the default value again! Differential Revision: https://phab.mercurial-scm.org/D10796

revlog: use entry in revlogv0.py This make things clearer and we can reply on some of the default value directly. Differential Revision: https://phab.mercurial-scm.org/D10795

revlog: use the new `entry` function in revlog.py This just make the construction of the tuple clearer. Differential Revision: https://phab.mercurial-scm.org/D10794

revlog: add a function to build index entry tuple Keeping index entry as tuple make sense for performance reason, however it does not means we need to manually build that tuple for all piece of python code that are not performance critical. So we add a nice function responsible to build the tuple using argument using explicit keyword argument. Differential Revision: https://phab.mercurial-scm.org/D10793

revlog: move `offset_type` to `revlogutils` This multiple module are using this so it make sense to move it at the utility level. Differential Revision: https://phab.mercurial-scm.org/D10792

revlog: move entry documentation alongside new related constants Accessing individual index-entry element is usually done using integer directly. This is presumably for "performance reasons". However as the index entry gain more and more element it seems useful to get the option to use symbolic constant to access item, for both clarify and versatility. We will probably keep using integer for performance critical path, but less critical code may start using them now that they are declared. Differential Revision: https://phab.mercurial-scm.org/D10791

censor: drop size limitation on the tombstone This limitation seems to date back from a previous implementation of censors, were the revision were censored" in place. For a long while the implementation actually involved rewriting the revlog before replacing the original data. So it looks like we can safely remove this limitation. The tests suite agrees. Differential Revision: https://phab.mercurial-scm.org/D10790

revlog: move censoring code in a dedicated module This code is quite specific and we are about to add more of it for revlog-v2 (and other derived version). So we move this code in a dedicated module in `mercurial/revlogutils/`. This looks like a good fit. The diff is huge because I used `hg copy` to create the new file so that we keep the history of the censor code. Differential Revision: https://phab.mercurial-scm.org/D10789

censor: reduce risk of censor test blowing up output limit The censored file is really big, and we keep using `hg cat` on it, with the assumption that it is censored. However, when it is not censored, the amount of extra line output is huge and quickly saturate the backlog of terminal and CI system. Differential Revision: https://phab.mercurial-scm.org/D10788

revlog: store sidedata in their own file This makes sidedata manipulation simpler and results in more compact data when traversing either data or sidedata. Differential Revision: https://phab.mercurial-scm.org/D10787

revlog: fix docket.date_filepath docstring This was copy-pasted too quickly. Differential Revision: https://phab.mercurial-scm.org/D10786

revlog: use dedicated code for reading sidedata We are about to introduce a new, dedicated, file to store sidedata. Before doing so, we make sidedata reading go through different code as reading data chunk. This will simplify some of the complexity of the next changesets. The reading is very simple right now and will need some improvement later to reuse some of the caching strategy we use for the data file. Differential Revision: https://phab.mercurial-scm.org/D10785

revlog: simplify "partial read" error message We are about to reuse this message in more places and the current handling is pretty hard to read. So we eat two pie with one stone and clean up this. Differential Revision: https://phab.mercurial-scm.org/D10784

revlog: no longer return sidedata from `_revisiondata` All users have been migrated. When they was introduced sidedata where grouped with the actual revision data and unpacking one came with the other. Sidedata moved be stored "independently" and it no longer make sense to retrieve both at the same time unconditionnaly. Differential Revision: https://phab.mercurial-scm.org/D10783

revlog: use `self.sidedata` directly in `revlog.clone` We want to change `revlog._revisiondata` signature, so we need to migrate users away. Differential Revision: https://phab.mercurial-scm.org/D10782

revlog: use `self.sidedata` directly to construct changelogrevision We want to change `revlog._revisiondata` signature, so we need to migrate users away. Differential Revision: https://phab.mercurial-scm.org/D10781

revlog: implement sidedata without using _revisiondata When they was introduced sidedata where grouped with the actual revision data and unpacking one came with the other. Sidedata moved be stored "independently" and it no longer make sense to retrieve both at the same time unconditionnaly. We start with changeset the implementation of the `revlog.sidedata` command to no longer use `self._revisiondata`. More users need to be migrated to direct usage of this `revlog.sidedata` method. This will be done in the coming changesets. Differential Revision: https://phab.mercurial-scm.org/D10780

revlogv2: drop "sidedatacompression" from the list of todos This was implemented in c4dbb7636a12 and 07641bafa646, but I forgot to drop it from the todo-list. Differential Revision: https://phab.mercurial-scm.org/D10779

rust: Use a maintained crate for SHA-1 hashing https://crates.io/crates/rust-crypto hasn’t been updated in 5 years. This doesn’t neccesarily mean there’s anything wrong with it, but if something comes up it’s preferable to rely on libraries that have active maintainers. Use https://crates.io/crates/sha-1 from https://github.com/RustCrypto instead Differential Revision: https://phab.mercurial-scm.org/D10835

rust: Parse "subinclude"d files along the way, not later When parsing a `.hgignore` file and encountering an `include:` line, the included file is parsed recursively right then in a depth-first fashion. With `subinclude:` however included files were parsed (recursively) much later. This changes it to be expanded during parsing, like `.hgignore`. The motivation for this is an upcoming changeset that needs to detect changes in which files are ignored or not. The plan is to hash all ignore files while they are being read, and store that hash in the dirstate (in v2 format). In order to allow a potential alternative implementations to read that format, the algorithm to compute that hash must be documented. Having a well-defined depth-first ordering for the tree of (sub-)included files makes that easier. Differential Revision: https://phab.mercurial-scm.org/D10834

rust: Make some file path parameters less generic These are not widely used APIs that benefit from being maximally flexible, taking an explicit `&Path` borrow is fine and simplifies their internals. Differential Revision: https://phab.mercurial-scm.org/D10833

convert-bazaar: use breezy package instead of old bzr one Breezy is the most recent of the two, and works on Python 3 while being compatible with the (old) Bazaar file format. This patch contains a variety of unicode <-> bytes changes, API breakage fixing, restoring failing imports and changing the executable from `bzr` to `brz`. I recommend using the debian packages for `brz` and `python3-breezy` (3.1+), because the pip package seems to be haunted by radioactive dragons. Differential Revision: https://phab.mercurial-scm.org/D10513

bzr: only support bzr conversion on Python3 Python 2 support will be dropped from Mercurial before anyone needs to do a bzr -> hg conversion on Python 2 again. Bazaar tests were broken with the new lib anyway, which we'll get into in the next patches. Differential Revision: https://phab.mercurial-scm.org/D10512

rewriteutil: add pointer to help text when rewrite would cause divergence The evolve extension's version of the hint has this pointer. I missed it when I moved it to core. This is pretty much the same as D10723. Turns out I missed two places :( Differential Revision: https://phab.mercurial-scm.org/D10832

rhg: Add support for dirstate-v2 Differential Revision: https://phab.mercurial-scm.org/D10804

rhg: Remove some intermediate Vecs in `rhg files` Instead of calling `parse_dirstate` which then calls `parse_dirstate_entries`, call the latter directly in order to skip some intermediate steps. Differential Revision: https://phab.mercurial-scm.org/D10803

merge: make applyupdates() not mutate mresult argument We have an extension at work that overrides `merge.applyupdates()` to make it skip some writes and instead instruct the virtual filesystem we use to get a different version. That override doesn't work correctly when doing `hg co -m` and there's a modified file in the dirstate that's deleted in the destination. That's because `applyupdates()` mutates its `mresult` argument and our extension had passed in a modified copied of `mresult` to the overridden function, which resulted in the mutation not having any effect. This patch fixes that by letting the caller (i.e. `merge._update()`) update `mresult` with the extra actions instead. Besides fixing our internal extension, that seems cleaner to me anyway (better to not mutate `mresult` only in some cases and we can skip some of the logic if we're not going to update the dirstate anyway). Differential Revision: https://phab.mercurial-scm.org/D10830

index: use an explicit constant for INDEX_HEADER format and use it for docket This avoid leaking python-3.6 compatibility details too much. Differential Revision: https://phab.mercurial-scm.org/D10831

verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. Differential Revision: https://phab.mercurial-scm.org/D10822

verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. We extract the long message in a module level constant for clarity. Differential Revision: https://phab.mercurial-scm.org/D10821

verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. Differential Revision: https://phab.mercurial-scm.org/D10820

verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. We extract the long message in a module level constant for clarity. Differential Revision: https://phab.mercurial-scm.org/D10819

verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. Differential Revision: https://phab.mercurial-scm.org/D10818

verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. Differential Revision: https://phab.mercurial-scm.org/D10817

verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. Differential Revision: https://phab.mercurial-scm.org/D10816

verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. We extract the long message in a module level constant for clarity. Differential Revision: https://phab.mercurial-scm.org/D10815

verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. Differential Revision: https://phab.mercurial-scm.org/D10814

verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. Differential Revision: https://phab.mercurial-scm.org/D10813

verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. Differential Revision: https://phab.mercurial-scm.org/D10812

verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. Differential Revision: https://phab.mercurial-scm.org/D10811

verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. We extract the long message in a module level constant for clarity. Differential Revision: https://phab.mercurial-scm.org/D10810

verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. Differential Revision: https://phab.mercurial-scm.org/D10809

verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. Differential Revision: https://phab.mercurial-scm.org/D10808

verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. Differential Revision: https://phab.mercurial-scm.org/D10807

verify: expand a one liner into explicit commands The result is not longer, but it is more edible. Differential Revision: https://phab.mercurial-scm.org/D10806

verify: align a comment line This argument description is back with it comrade. Differential Revision: https://phab.mercurial-scm.org/D10805

dirstate-v2: Drop parent directory cache when removing a dirstate node The premise of the directory cache is that the dirstate contains child nodes for every entry that `read_dir` would return. When removing nodes, that may not be the case anymore so the cache should be invalidated. Differential Revision: https://phab.mercurial-scm.org/D10829

dirstate-v2: Add --dirs to debugdirstate command `hg debugdirstate --dirs` also shows information stored in the dirstate (for `read_dir` caching) about directories. Differential Revision: https://phab.mercurial-scm.org/D10828

dirstate-v2: Write .hg/dirstate back to disk on directory cache changes Differential Revision: https://phab.mercurial-scm.org/D10827

dirstate-v2: Skip readdir in status based on directory mtime When calling `read_dir` during `status` and the directory is found to be eligible for caching (see code comments), write the directory’s mtime to the dirstate. The presence of a directory mtime in the dirstate is meaningful and indicates eligibility. When an eligible directory mtime is found in the dirstate and `stat()` shows that the mtime has not changed, `status` can skip calling `read_dir` again and instead rely on the names of child nodes in the dirstate tree. The `tempfile` crate is used to create a temporary file in order to use its modification time as "current time" with the same truncation as other files and directories would have in their own modification time. Differential Revision: https://phab.mercurial-scm.org/D10826

dirstate-v2: Allow tree nodes without an entry to store a timestamp Timestamps are stored on 96 bits: * 64 bits for the signed number of seconds since the Unix epoch * 32 bits for the nanoseconds in the `0 <= ns < 1_000_000_000` range For now timestamps are not used or set yet. Differential Revision: https://phab.mercurial-scm.org/D10825

dirstate-tree: Change status() results to not borrow DirstateMap The `status` function takes a `&'tree mut DirstateMap<'on_disk>` parameter. `'on_disk` borrows a read-only byte buffer with the contents of the `.hg/dirstate` file. `DirstateMap` internally uses represents file paths as `std::borrow::Cow<'on_disk, HgPath>`, which borrows the byte buffer when possible and allocates an owned string if not, such as for files added to the dirstate after it was loaded from disk. Previously the return type of of `status` has a `'tree` lifetime, meaning it could borrow all paths from the `DirstateMap`. With this changeset, that lifetime is changed to `'on_disk` meaning that only paths from the byte buffer can be borrowed, and paths allocated by `DirstateMap` must be copied. Usually most paths are in the byte buffer, and most paths are not part of the return value of `status`, so the number of extra copies should be small. This change will enable `status` to mutate the `DirstateMap` after it has finished constructing its return value. Previously such mutation would be prevented by possible on-going borrows. Differential Revision: https://phab.mercurial-scm.org/D10824

dirstate-tree: Fix status algorithm with unreadable directory When reading a directory fails such as because of insufficient permissions, it should be treated as empty by status instead of skipped entirely. Differential Revision: https://phab.mercurial-scm.org/D10823

docket: make compatible with py3.6, where Struct.format is bytes Differential Revision: https://phab.mercurial-scm.org/D10770

packaging: disable rust extensions again on CentOS Backed out changeset eccbfa7e19c0 We're seeing (very rarely) crashes of 'hg purge' on some of our machines (see https://bz.mercurial-scm.org/show_bug.cgi?id=6509 ). Unfortunately, I haven't been able to find out much more about what is going wrong. To avoid further impact on our users and CI, I would prefer to disable the rust extensions for now. Differential Revision: https://phab.mercurial-scm.org/D10877

cext: fix memory leak in phases computation Without this a buffer whose size in bytes is the number of changesets in the repository is leaked each time the repository is opened and changeset phases are computed. Impact: the current code in hgwebdir creates a new `localrepository` instance for each HTTP request. Since any pull or push is made of several requests, a team of 100 people can easily produce thousands of such requests per day. Being a low-level malloc, this leak can't be seen with the gc module and tools relying on that, but was spotted by valgrind immediately. Reproduction ------------ for i in range(cl_args.iterations): repo = hg.repository(baseui, repo_path) rev = repo.revs(rev).first() ctx = repo[rev] del ctx del repo # avoid any pollution by other type of leak # (that should be fixed in 5.8) repoview._filteredrepotypes.clear() gc.collect() Measurements ------------ Resident Set Size (RSS), taken on a clone of mozilla-central for performance analysis (440 000 changesets). before: 5.8+hg19.5ac0f2a8ba72 1000 iterations: 1606MB 5.8+hg19.5ac0f2a8ba72 10000 iterations: 5723MB after: 5.8+hg20.e2084d39e145 1000 iterations: 555MB 5.8+hg20.e2084d39e145 10000 iterations: 555MB (double checked, not a copy/paste error) (e2084d39e14 is the present changeset, before amendment of the message to add the measurements)

revlogv2: make sure bundling pick a compatible bundle format Before this change, revlog-v2 repository where bundled using the incompatible "v1" format. Differential Revision: https://phab.mercurial-scm.org/D10802

censor: do not process sidedata of censored revision while bundling The revision is censored, we should ignore it. Differential Revision: https://phab.mercurial-scm.org/D10801

changegroup: fix deltachunk API to be consistent from one class to another Depending of the subclass the 8th index of `chunkdata` items was either a sidedata dict of a proto_flags integer. We have not fixed the inconsistency and we already return fixed "delta" items from `deltaiter`. Differential Revision: https://phab.mercurial-scm.org/D10778

fuzz: add hg to sys.path when constructing mpatch corpus Differential Revision: https://phab.mercurial-scm.org/D10777

dirstate-tree: Skip readdir() in `hg status -mard` When running the status algorithm in a mode where we don’t list unknown or ignored files, all we care about are files that are listed in the dirstate. We can there for skip making expensive calls to readdir() to list the contents of filesystem directories, and instead only run stat() to get the filesystem state of files listed in the dirstate. (This state may be an error for files that don’t exist anymore on the filesystem.) On 16 CPU threads, this reduces the time spent in the `status()` function for `hg status -mard` on an old snapshot of mozilla-central from ~70ms to ~50ms. Differential Revision: https://phab.mercurial-scm.org/D10752

dirstate-v2: Parse the dirstate lazily, with copy-on-write nodes TODO: more description Differential Revision: https://phab.mercurial-scm.org/D10751

dirstate-v2: Make the dirstate bytes buffer available in more places Differential Revision: https://phab.mercurial-scm.org/D10750

dirstate-v2: Make more APIs fallible, returning Result When parsing becomes lazy, parse error will potentially happen in more places. This propagates such errors to callers. Differential Revision: https://phab.mercurial-scm.org/D10749

dirstate-v2: Add a zero-size error type for dirstate v2 parse errors This error should only happen if Mercurial is buggy or the file is corrupted. It indicates for example that: * A part of the file refers to another part, and the byte offset or item count would cause reading out of bounds, beyond the end of the file. * The byte for an entry state has an invalid value When parsing becomes lazy, many more functions will return a `Result` with this error. Making it zero-size reduces the work that the `?` operator needs to do to pass around the error value. Differential Revision: https://phab.mercurial-scm.org/D10748

dirstate-tree: Add `NodeRef` and `ChildNodesRef` enums They are used instead of `&Node` and `&ChildNodes` respectively. The `ChildNodes` type alias also becomes a similar enum. For now they only have one variant each, to be extended later. Adding enums now forces various use sites go through new methods instead of manipulating the underlying data structure directly. Differential Revision: https://phab.mercurial-scm.org/D10747

rust: Return owned instead of borrowed DirstateEntry in DirstateMap APIs This will enable the tree-based DirstateMap to not always have an actual DirstateEntry in memory for all nodes, but construct it on demand. Differential Revision: https://phab.mercurial-scm.org/D10746

dirstate-tree: Downgrade `&mut Node` to `&Node` in status and serialization Mutable access is not used, and upcoming changes will make it more costly (with copy-on-write nodes that can be read from disk representation) Differential Revision: https://phab.mercurial-scm.org/D10745

dirstate-tree: Remove DirstateMap::iter_node_data_mut In an upcoming changeset we want DirstateMap to be able to work directly with nodes in their "on disk" representation, without always allocating corresponding in-memory data structures. Nodes would have two possible representations: one immutable "on disk" refering to the bytes buffer of the contents of the .hg/dirstate file, and one mutable with HashMap like the curren data structure. These nodes would have copy-on-write semantics: when an immutable node would need to be mutated, instead we allocate new mutable node for it and its ancestors. A mutable iterator of the entire tree would still be possible, but it would become much more expensive since we’d need to allocate mutable nodes for everything. Instead, remove this iterator. It was only used to clear ambiguous mtimes while serializing the `DirstateMap`. Instead clearing and serialization are now two separate passes. Clearing first uses an immutable iterator to collect the paths of nodes that need to be cleared, then accesses only those nodes mutably. Differential Revision: https://phab.mercurial-scm.org/D10744

merge with stable

revlog: close the index file handle after the data one This make sure the data file is flushed before the index. preventing the index to reference unflushed data. Differential Revision: https://phab.mercurial-scm.org/D10776

revlog: simplify the try nesting in the `_writing` context Lets use a single try, with conditional cleanup. This make is easier to add a file handle dedicated to sidedata. Differential Revision: https://phab.mercurial-scm.org/D10775

revlogv2: add a `get_data` helper to grab the next piece of docket This make the processing more compact but abstracting repetitive processing away. Differential Revision: https://phab.mercurial-scm.org/D10774

revlogv2: simplify and clarify the processing of each entry As we add more entries and some of them has non trivial processing it seems useful to make the processing leaner and clearly separated to simplify futures patches. Differential Revision: https://phab.mercurial-scm.org/D10773

revlogv2: use a unique filename for data Having a unique data will allow for ambiguity less rewriting of revlog content, something useful to clarify handling of some operation like censoring or stripping. Differential Revision: https://phab.mercurial-scm.org/D10772

revlogv2: use a unique filename for index Having a unique index will allow for ambiguity less rewriting of revlog content, something useful to clarify handling of some operation like censoring or stripping. Differential Revision: https://phab.mercurial-scm.org/D10771

rhg: A missing .hg/dirstate file is not an error Instead treat it as like an empty file Differential Revision: https://phab.mercurial-scm.org/D10766

rhg: Sort `rhg status` output correctly * The relative order of states is: modified, added, removed, deleted, unknown, ignored, clean * Files in the same state should be sorted by name, regardless of whether or not the were in "unsure" state based on metadata alone. Differential Revision: https://phab.mercurial-scm.org/D10765

dirstate-v2: Add `hg debugupgraderepo` command support This command changes changes the file formats used inside an existing repository to what they would be in a new repository with the current config. For example: hg debugupgraderepo --config format.exp-dirstate-v2=1 --run hg debugupgraderepo --config format.exp-dirstate-v2=0 --run If a repository has a dirstate in v1 format, the first command would upgrade it to dirstate-v2. Conversely, if a repository has a dirstate in v2 format, the second command would downgrade it to v1. (Both may also run some unrelated upgrades.) Since `format.exp-dirstate-v2` is currently disabled by default, not specifying it in `--config` or any configuration file would result in the second command. Differential Revision: https://phab.mercurial-scm.org/D10769

upgrade: Use `improvement` subclasses everywhere, not instances This changes the source definition of optimizations to match that of formats: a subclass with a decorator, instead of an instance passed to a function call. Not having any instance removes the confusion between class attributes and instance attributes, which were used interchangeably. Differential Revision: https://phab.mercurial-scm.org/D10768

tests: monkeypatch `util.get_password()` to avoid deadlocks on Windows This should have been part of 5b3513177f2b. Differential Revision: https://phab.mercurial-scm.org/D10759

rust-status: fix ignore and include not composing (issue6514) While the fix is pretty simple, the overall dispatch logic has become kind of ugly. Thankfully we're currently upstreaming a better algorithm, this code is temporary anyway. Differential Revision: https://phab.mercurial-scm.org/D10639

rust-status: highlight a bug in Rust-augmented status This was reported in issue6514, confirmed with this test reproduction. This will be fixed with the next changeset. Differential Revision: https://phab.mercurial-scm.org/D10638

exewrapper: avoid directly linking against python3X.dll Subsequent code calls `LoadLibrary()` to attempt to load the DLL, but because of this symbol reference, there is an attempt to load the DLL used during the build prior to `_main()` running. This causes the whole process to fail if the DLL isn't in the standard search path. That also means it will never load the DLL for HackableMercurial. (Maybe we should get rid of that for py3, since you can install python for a user without admin rights?) This could also be resolved by calling `GetProcAddress()` on the symbol and dereferencing it, but using the environment variable is consistent with the *.bat file since fc8a5c9ecee0. (The environment variable persists after the interpreter is initialized.) Far more concerning is somehow I've gotten my system into a state where setting the flag causes any output to the pager to be lost (as if it wasn't set at all) in MSYS, cmd.exe, WSL, and PowerShell using py3.9.0, but the environment variable works properly. I'm sure this flag worked on some versions of py3, so I'm not sure what's going on here. This is might be related to init config related changes in 3.8[1], since it works with 3.7.8, but fails with 3.8.1. Somebody who understands encoding issues better than I do should give some thought to if we need to make some changes to our encoding strategy on Windows with py3. With or without the flag/envvar, there is proper output if the command is directly paged by piping to `more.com` (in any environment) or `less` (in MSYS and WSL), or if paging is disabled with `--pager=no`. Legacy mode is required though when Mercurial decides to spin up a pager. [1] https://bugs.python.org/issue41941 Differential Revision: https://phab.mercurial-scm.org/D10756

tests: partially fix test-transaction-rollback-on-revlog-split.t on Windows For whatever reason, the extra space causes different output on Windows: $ printf '% 20d' '1' | xxd - 0000000: 2020 2020 2020 2020 2020 2020 2020 2020 0000010: 2020 2020 31 1 vs WSL: /mnt/c/Users/Matt/hg/tests$ printf '% 20d' '1' | xxd - 00000000: 2020 2020 2020 2020 2020 2020 2020 2020 00000010: 2020 2031 Without the space, they both agree the last word is 2031. There are other failures on Windows that I don't understand that are introduced by 46b828b85eb7 and 18415fc918a1. Differential Revision: https://phab.mercurial-scm.org/D10763

tests: fix test-transaction-safety.t on Windows Mostly this is about `C:\dir\sub` style paths being turned into `C:dirsub` by MSYS, and not being about to execute shell scripts in hooks directly. See also a12819559ccb::9e3979a25bfe. Differential Revision: https://phab.mercurial-scm.org/D10762

docket: move the uid logic in the `revlogutils.docket` module We want to use it for revlog-v2 (& Co), it seems more logical to have the logic lives inside the `docket` file than the `nodemap` file. Differential Revision: https://phab.mercurial-scm.org/D10755

persistent-nodemap: use the intended uuid size I overlooked this in the initial implementation. The format already supporting any uuid size so, nothing horrible should happens. Differential Revision: https://phab.mercurial-scm.org/D10754

persistent-nodemap: add a way to make the picked uid predictable This will make the test less volatile and therefore easier to maintain. I am setting this is for the persistent nodemap docket first, but my target is to use similar logic for the revlogv2 (and co) test. For persisent nodemap, they are too many different in how each test variants (with rust, without rust, etc) needs new uid for this be really useful (right now). However it will be directly useful for the revlogv2 logic. Differential Revision: https://phab.mercurial-scm.org/D10753

run-tests: disable color in 'auto' mode on Windows if stdout is redirected Differential Revision: https://phab.mercurial-scm.org/D10761

run-tests: enable color on Windows In setting up the CI for Windows on heptapod, I noticed it was complaining about color not being enabled because pygments wasn't installed- even though it was. I had initially disabled color on Windows when using Windows 7, because that didn't understand ANSI color codes and made a mess of the output. But now that it's been unsupported for over a year, I don't think we should care about it either. It's admittedly a hack to depend on Mercurial proper to enable color support in the terminal, but I didn't feel like duplicating that code. I'm under the impression that 3rd party stuff is supposed to use this runner in the Mercurial repo instead of using their own copy, so I think it's safe to assume the Mercurial code is available. If it's not, it won't break anything. Differential Revision: https://phab.mercurial-scm.org/D10760

transaction: trivial refactoring Differential Revision: https://phab.mercurial-scm.org/D10714

test: remove some unnecessary dependency on repo format Differential Revision: https://phab.mercurial-scm.org/D10757

hg-cpython: fix new occuring TypeError dd339191f2dc introduced a minor refactoring of error types that highlighted a fragile error creation mechanism that was in place in the Rust `MixedIndex`. `PyErr::from_instance` also accepts a Python class, as long as it's an `Exception` class. Before the aforementioned commit, we never ran into a case where this duck-typing mechanism failed. We rectify this behavior by doing the instantiation ourselves. Differential Revision: https://phab.mercurial-scm.org/D10764

errors: make exit codes class variables instead Kyle pointed out to me that we can simply make the exit codes class variables. Python provides some magic for making them accessible as instance variables. This also makes it easier to let subclasses of existing errors override the exit codes by letting them simply define their own values as class variables. That means that there's no need to pass them into the superclass's constructor arguments, so the superclass doesn't need to expose the them as arguments. (Making a subclass set a different exit code for a subclass of `StorageError` was actually the goal with my recent series.) Differential Revision: https://phab.mercurial-scm.org/D10758

narrow: widen when files are excluded by sparse and not included by narrow In a repo where some directories are included by narrow and the complement are excluded by sparse, it was previously impossible to widen either because trying to widen narrow would complain that the requested files are outside the sparse checkout and trying to widen sparse would complain that the requested files are outside the narrow checkout. This changes the `hg tracked --addinclude` command to only actually update any newly accessible files in the dirstate if they are also accessible via sparse. Differential Revision: https://phab.mercurial-scm.org/D10734

hghave: make error output easier to diagnose I had a typo that meant the new bash check was throwing an exception (due to a missing argument), but it was very hard to diagnose without this change. Differential Revision: https://phab.mercurial-scm.org/D10733

tests: add req on bash for test-transaction-rollback-on-sigpipe (issue6429) I think we could work around this by rewriting the helper scripts in Python, but I don't want to deal with that now and this should prevent failures due to a lack of bash. Differential Revision: https://phab.mercurial-scm.org/D10732

updatecaches: deprecate the `full` argument Now that all users were migrated, we can use deprecate the old way. This would give potential extensions code a heads up on the API change. Differential Revision: https://phab.mercurial-scm.org/D10731

updatecaches: use the `caches` argument instead of a special `full` value After a clone we want to update most cachem, but not exactly all of them. We can now cleanly express this. Differential Revision: https://phab.mercurial-scm.org/D10730

updatecaches: use the caches argument in `hg debugupdatecaches` This is the new way. Differential Revision: https://phab.mercurial-scm.org/D10729

updatecaches: adds a `caches` parameters to `repo.updatecaches` It will superseed the `full` parameters (and its `post-clone` variant from stable). Various caller will be updated in the rest of this series. Differential Revision: https://phab.mercurial-scm.org/D10728

updatecaches: introduce a set of constants to control which are updated Passing around a set of constant to select what need warming will be cleaner and more flexible. We did not changed the API yet, as this changes is already large enough. In the rest of the rest we will change more code to actually use this constants (or more realistically pre-defined set of constant directly) Differential Revision: https://phab.mercurial-scm.org/D10727

errors: make StorageError subclass Error, attaching an exit code to it Differential Revision: https://phab.mercurial-scm.org/D10741

revlog: avoid raising no-arg RevlogError for internal flow control I'm about to make RevlogError require a `message` argument and this code was failing. This patch refactors it to not raise an exception for intra-function flow control. Differential Revision: https://phab.mercurial-scm.org/D10740

errors: catch the new Error class in scmutil and chgserver Differential Revision: https://phab.mercurial-scm.org/D10739

errors: create superclass for Abort exception I'd like to let extensions subclass `StorageError` to define a custom exit code. However, `StorageError` does not extend `Abort` (which is where the exit code currently lives), and it seems that it's not supposed to either (`StorageError` seems to be for lower-level errors and `Abort` is for command-level errors). This patch therefore extracts all the code from `Abort` into a new `Error` class, which I'll soon make `StorageError` also extend. Differential Revision: https://phab.mercurial-scm.org/D10738

errors: make InterventionRequired subclass Abort The docstring for `Abort` says that it's for errors raised by commands and `InterventionRequired` is definitely something raised by commands, so it seems that it should be an `Abort`. This patch makes it so. It adds a `coarse_exit_code` (in addition to the already existing `detailed_exit_code`) to `Abort` to achieve that, since `InterventionRequired` should result in a special exit code even when the `ui.detailed-exit-code` config is not set. Differential Revision: https://phab.mercurial-scm.org/D10737

errors: move Abort earlier, so more exceptions can subclass it I'd like to make at least `InterventionRequired` subclass `Abort` and Python requires the superclass to be defined before the subtype. Differential Revision: https://phab.mercurial-scm.org/D10736

errors: let each Abort subclass define its error code It's more flexible to have the error codes defined on the error types themselves. That way extensions can easily set their own exit code. It also means that we can reduce a bit of duplication betwen `scmutil.callcatch()` and `chgserver.chgcmdserver.validate()`. Differential Revision: https://phab.mercurial-scm.org/D10735

rust: Fix "panic message is not a string literal" warnings These deprecation warnings would not become errors until we actively port crates to the (not yet released) Rust 2021 edition, but fixing them anyway reduces console output noise. Differential Revision: https://phab.mercurial-scm.org/D10743

rust: Add type annotation to fix inference on Rust Nightly When compiling with Rust Nightly, the im-rs crate silently makes use of the experimental language feature for trait impl specialization. This apperently changes public its APIs in subtle ways such that type inference of some user code can fail where it succeeds when specialization is disabled. This made Mercurial’s Rust unit tests have compilation errors on Nightly. I have not managed to find the exactl root cause, but I wrote down my findings so far at https://github.com/bodil/im-rs/issues/188 This adds type annotation to make unit tests rely less on type inference and work around the issue. Differential Revision: https://phab.mercurial-scm.org/D10742

recover: only apply last journal record per file (issue6423) This got broken in 2019 when the size check was introduced. It is most noticable when dealing with transactions that involve an inline to non-inline revlog storage transaction. It wasn't seen as much at the time because the in-memory journal actually de-duplicated the entry implicity, but since 63edc384d3b7 the on-disk journal is used for rollback as well as recover. Differential Revision: https://phab.mercurial-scm.org/D10726

revlog: update data file record before index rename When migrating from inline to non-inline data storage, the data file is recorded initially as zero sized so that it is removed on failure. But the record has to be updated before the index is renamed, otherwise data is lost on rollback. Differential Revision: https://phab.mercurial-scm.org/D10725

revlog: fix index computation during inline->non-inline transition The computation in 63edc384d3b7 failed to factor in the index entries themselve as revlog.start() doesn't count them. Found by Valtenin Gatienbaron with a more precise test case from me. Differential Revision: https://phab.mercurial-scm.org/D10724

dirstate-v2: Change the on-disk format to be tree-shaped Nodes are stored not only for tracked files but also for their ancestor directories. A node has "pointers" (byte count from the start of the file) to its direct child nodes. Everything can be accessed with zero copy. Differential Revision: https://phab.mercurial-scm.org/D10722

dirstate-tree: Extract into a method sorting children of a given node A later changset will use this in another place. This is an associated function (that Python would call static method) instead of a free function so it doesn’t need to be imported separately. It’s on `Node` rather than `ChildNodes` because the latter is a type alias to an external type (`HashMap`) so that would require an extension trait which needs to be imported separately. Differential Revision: https://phab.mercurial-scm.org/D10721

dirstate-v2: Add a variant of some tests, that uses the new format With this, the new format receives some testing every time someone runs tests with Rust extensions enabled, including on CI. Differential Revision: https://phab.mercurial-scm.org/D10720

dirstate-v2: Change the on-disk format when the requirement is enabled For now, the format is the same except with an additional marker at the start. This marker is redundant: for existing repositories it is `.hg/requires` that determines which format to use. For new repositories, it is the new `format.exp-dirstate-v2` config. There is no upgrade or downgrade so far. Most of the changes are about plumbing a boolean through layers of APIs to indicate which format should be used. Differential Revision: https://phab.mercurial-scm.org/D10719

dirstate-v2: Update the expected output of some tests for new requirement Fix most test failures (except in test-narrow-debugrebuilddirstate.t and test-upgrade-repo.t) caused by the new entry in config or in .hg/requires when running `run-tests.py --extra-config-opt format.exp-dirstate-v2=1` There is no CI so far for this configuration. Differential Revision: https://phab.mercurial-scm.org/D10718

tests: More cleanly separate expected hexdump output There are more lines that differ than are in common, and dirstate-v2 will complicate that further. Differential Revision: https://phab.mercurial-scm.org/D10717

dirstate-v2: Add a new experimental `exp-dirstate-v2` repository requirement This requirement is added to `.hg/requires` when creating a new repository if Rust extensions are enabled and the `format.exp-dirstate-v2` config is set. Nothing yet changes based on this requirement, but its mere presence affects some tests (for example if they print `.hg/requires`). The next two changesets update tests’ expected outputs accordingly. There is no CI so far that enables this configuration. Differential Revision: https://phab.mercurial-scm.org/D10716

urlutil: provide some information about "bad url" when processing `pushurl` It appears pushurl only support `<proto>://` entries. This is not obvious and can lead to obscure error. We make the error less obscure as a start.. Differential Revision: https://phab.mercurial-scm.org/D10455

revset: document the `outgoing` behavior if the path resolve to multiple urls Differential Revision: https://phab.mercurial-scm.org/D10454

multi-urls: document the feature We have a feature lets make it visible to people. Differential Revision: https://phab.mercurial-scm.org/D10453

multi-urls: add a boolean suboption that unlock path specification as list When this option is set, a list of patch can be specifed as value for `[paths]` entries. For the command who support it, this behave the same as providing multiple destination of the command line. Differential Revision: https://phab.mercurial-scm.org/D10452

urlutil: move url "fixing" at the time of `ui.paths` initialization Doing such fixing at the time is simpler and will be necessary to deal with urls list in a sane manner. It also reduce the size of fix-config which is always better. I wish we could get ride of the hackish way to pass the root around, I suspect that the `root` variable could be stored as part of the config value, along side the source. However getting to the end of this `root` business is a far too large detours to make now. The test change to `tests/test-hgrc.t` and `test-config.t` are expectied since we are not longer altering the config itself, but the way it is interpreted when building path. This seems more correct. I also added a couple of test call to `test-config.t` and `test-globalopts.t` to clarify that the expanding process is properly happening a the right time. Differential Revision: https://phab.mercurial-scm.org/D10451

rewriteutil: add pointer to help text when rewrite would cause divergence The evolve extension's version of the hint has this pointer. I missed it when I moved it to core. Differential Revision: https://phab.mercurial-scm.org/D10723

changelogv2: use a dedicated on disk format for changelogv2 We drop two unused entry. This is mostly a proof of concept before starting to actually rework the format. Differential Revision: https://phab.mercurial-scm.org/D10667

revlog: do not call Rust code if the index is not compatible with it This will avoid hitting the TypeError we defined in the previous changesets. This is the simplest fix but not the most elegant. Ideally we would teach the Rust code to use any kind of revlog. However this is an adventure for another time. Differential Revision: https://phab.mercurial-scm.org/D10666

revlog: signal which revlog index are compatible with Rust Otherwise, Rust may treat python object like `cindex` object, leading to trouble. The new attribute is an integer because I expect we might need a flag field in the future. As a start we get the rust code to raise a clear TypeError. We will use the information in a smarter way in the next changesets. Differential Revision: https://phab.mercurial-scm.org/D10665

revlog: pass around the `rev` we deal with when packing/unpacking entry This will help code to compute some flag on the fly. Differential Revision: https://phab.mercurial-scm.org/D10664

changelogv2: use a dedicated version number If we want to change the format, we need a new version number. We start with that. Differential Revision: https://phab.mercurial-scm.org/D10663

changelogv2: `copies-side-data` now implies `changelogv2` To store information about file changes and copies only requires sidedata support for the changelog. So we only enables `changelogv2`. This is less impactful and should help us to ship the changeset centric copy tracing sooner. Differential Revision: https://phab.mercurial-scm.org/D10662

changelogv2: allow upgrade from and to this format Differential Revision: https://phab.mercurial-scm.org/D10661

changelogv2: introduce a "changelogv2" feature Right now, this means using revlogv2, but only for the changelog. We will have the format more unique in future changesets. Differential Revision: https://phab.mercurial-scm.org/D10660

revlogv2: drop the code related to inlined revlogv2 Tevlogv2 will not use inlined revlog. As a result we can drop the associated code. Differential Revision: https://phab.mercurial-scm.org/D10659

revlogv2: no longer attempt to use inline for new revlog revlogv2 steered away for the inline feature. Differential Revision: https://phab.mercurial-scm.org/D10658

sidedata: use the "feature" to detect that sidedata are enabled This is more versatile than relying on the revlogv2 requirements. Differential Revision: https://phab.mercurial-scm.org/D10657

revlog: compress sidedata when doing "post-pull" sidedata update All path writing sidedata are now using compression (when appropriate). Differential Revision: https://phab.mercurial-scm.org/D10656

revlog: compress sidedata in `_writeentry` When appropriate we store the sidedata in a compressed form. Differential Revision: https://phab.mercurial-scm.org/D10655

revlog: apply compression mode while reading sidedata Right now, we only store things as PLAIN, but we are ready for some compression now. Future changeset will start using some. Differential Revision: https://phab.mercurial-scm.org/D10654

revlog: introduce a compression mode for sidedata in the revlog index We will use this for compression for the sidedata payload. Differential Revision: https://phab.mercurial-scm.org/D10653

revlog: implement a "default compression" mode The revlog docker is now storing a default compression engine. When a chunk use that compression, a dedicated mode is used in the revlog entry and we can directly route it to the right decompressor. We should probably make PLAIN and DEFAULT mode the only available mode for revlogv2, but this is something for later. Differential Revision: https://phab.mercurial-scm.org/D10652

revlog: add a `_get_decompressor` method This logic is non-trivial and we will need to reuse it. Differential Revision: https://phab.mercurial-scm.org/D10651

revlog: introduce a plain compression mode That mode is simple it means the chunk contains uncompressed data and can be used directly. Differential Revision: https://phab.mercurial-scm.org/D10650

revlog: use an intermediate variable for `data` in `_chunk` This will help us to dispatch this to the right decompressor. Differential Revision: https://phab.mercurial-scm.org/D10649

revlogv2: preserve the compression mode on disk The value is still the same and still not treated, but now it exists one disk. Differential Revision: https://phab.mercurial-scm.org/D10648

revlog: use `format_version` to dispatch different version in index_get This is semantically more correct. Differential Revision: https://phab.mercurial-scm.org/D10647

revlog: add a "data compression mode" entry in the index tuple That will make it possible to keep track of compression information in the revlog index, opening the way to more efficient revision restoration (in native code, but the python usage is already defeating performance work). We start with adding a new entry to the index tuple, using a value matching the current behavior. We will introduce storage and other value in later changesets. Differential Revision: https://phab.mercurial-scm.org/D10646

revlog: fix the `null_item` attribute for V0 This is not a proper index tuple. Differential Revision: https://phab.mercurial-scm.org/D10645

revlog: create a create `null_item` attribute for V0 This value is currently wrong, but at least it is now -obviously- wrong. Differential Revision: https://phab.mercurial-scm.org/D10644