revlog: add a new test file focussed on testing transactionally issue What we test here is currently also covered by `test-hooks.t`. However having our own test file allow for more focussed/deep testing and to cover more variant when relevant. So we create a new test file. Differential Revision: https://phab.mercurial-scm.org/D10626

revlogv2: store version information in the docket only Having it duplicated in the index was both useless and a risk of discrepancy. Differential Revision: https://phab.mercurial-scm.org/D10625

revlogv2: introduce a very basic docket file This is the first stone toward using a docket file in revlogv2. Right now the docket is very basic and only store the version number (which is -also- stored into the index file…) and the other files have fixed name. This new implementation break transactionally… but they are no test checking transactionally for revlogv2… So I take this as an opportunity to start small. They are no usage of revlogv2 outside of tests anyway. The docket keeps the `.i` naming used by previous version index to preserve a unique entry point. We could decide to use a different name and look it up first, or to fully rework this in a future "store" version. However that does not seems necessary right now. We will re-introduces transactionality (and associated testing…) in a later changesets. A long list of TODOs have been added to the relevant comment. Differential Revision: https://phab.mercurial-scm.org/D10624

branching: merge stable into default

revlogv2: mark revlogv2 as requires a full upgrade This was overlooked previously. Differential Revision: https://phab.mercurial-scm.org/D10623

revlog: unify flag processing when loading index The new code use a simple declaration to do centralised processing. This is clearer, shorter and less error prone. This will be especially useful as we plan to add a fourth format: changelog-v2. Differential Revision: https://phab.mercurial-scm.org/D10622

revlog: unify checks for supported flag The new code use a simple declaration to do centralised checking. This is clearer, shorter and less error prone. This will be especially useful as we plan to add a fourth format: changelog-v2. Differential Revision: https://phab.mercurial-scm.org/D10621

side-data: drop the associated config and requirements This is no longer and extensions of revlog v1, but a part of revlog v2. We no longer needs independent config and requirement management for that. Differential Revision: https://phab.mercurial-scm.org/D10620

revlog: fix capitalisation of an error We don't start error message with capital letters. Differential Revision: https://phab.mercurial-scm.org/D10619

test: no longer directly use the sidedata config and requirements They are on their way out, so we simply use revlogv2 in these cases. Some test does not requires explicite request for the revlogv2 format since the upgrade to a side-data-capable format is impled by `exp-use-copies-side-data-changeset`. Differential Revision: https://phab.mercurial-scm.org/D10618

sidedata: add a 'side-data' repository feature and use it Most code don't really care how sidedata support is added, but it needs to know if it is present. To achieve this. we use the `repo.features` attributes with a new dedicated features". Having such centralised information is more robust and will help us to remove the sidedata requirements in a later changesets. Differential Revision: https://phab.mercurial-scm.org/D10617

statichttp: add the missing `features` attribute This make statichttp closer to the interface it is "supposed" to follow. Differential Revision: https://phab.mercurial-scm.org/D10616

sidedata: use revlogv2 requirement in the test helper We are about to drop the requirement dedicated to sidedata (since revlogv2 will support them unconditionally and previous version will not.). To prepare this more we adapt the test code to not use that requirements. Differential Revision: https://phab.mercurial-scm.org/D10615

upgrade: display sidedata before upgrade This is a small gratuitous change that help making the test simpler to understand on its own. Differential Revision: https://phab.mercurial-scm.org/D10614

revlog: determine sidedata support based on the revlog version Revlog version "2" support sidedata, previous version does not. So lets make the initialization simpler. Differential Revision: https://phab.mercurial-scm.org/D10613

requirements: no longer drop `generaldelta` requirement with revlogv2 A repository could use a mix of revlogv1 and revlogv2, making the requirements still necessary. Overall we should move away from the "requirements" file being used a way to configure the repository and stick to it "what do you need to access this repository". However this is a wider work for another time. In addition the logic we just dropped was confusing the `hg debugformat` command, breaking the upgrade code and inconsistent (eg: `sparse-revlog` is also implied by `revlogv2`). Finally, multiple other config option would imply the use of the `revlogv2` requirements, without drop the `generaldelta` one, leading to more inconsistency. Differential Revision: https://phab.mercurial-scm.org/D10612

config: drop the `format.exp-revlogv2.2` option It is oddly named and not used at all. The one used by the code is still `experimental.revlogv2`. So we drop that one option for consistency. We move associate documentation to the actual option. Differential Revision: https://phab.mercurial-scm.org/D10611

revlog: only use the `_indexfp` method for read operation This will avoid "other" code to not overlook `_writing` usage. We introduces private method dedicated to writing to make use the right option are always used. Differential Revision: https://phab.mercurial-scm.org/D10610

revlog: use `_writing` in `rewrite_sidedata` Special cases aren't special enough to break the rules. Differential Revision: https://phab.mercurial-scm.org/D10609

revlog: open files in 'r+' instead of 'a+' The code doing actual writing is already doing the necessary seeking, so we could safely use 'r+'. This make the file objecs usable in more situation, like updating the sidedata information during pull. revlog: forcibly move the file cursor at the right location before writing This is a paranoid change in case the changelog computation moved the cursors under our feets. This is not known to happens right now. Differential Revision: https://phab.mercurial-scm.org/D10608

revlog: pass a transaction object to `rewrite_sidedata` The `_writing` context need one, so we update the function signature before considering using `_writing` in rewrite_sidedata. Differential Revision: https://phab.mercurial-scm.org/D10607

revlog: rename variable in `rewrite_sidedata` to match other code Let's call the index file object and `ifh` and the data file object `dfh` as the rest of the revlog code. This will make future change clearer. Differential Revision: https://phab.mercurial-scm.org/D10606

revlog: introduce a mandatory `_writing` context to update revlog content Before this change, various revlog methods where managing the opening and closing of the revlog files manually and passing the file descriptor alors the call path. To simplify the tracking of the write operation by a future docket, we need something more organised. As a result, we introduce a `revlog._writing` context manager that will wrap each revlog update operation. The file descriptor are kept in the existing `revlog._writinghandles` parameter that was already used by the `addgroup` logic. All this change is internal to the revlog only, the "public" interface is not affected. The `addrevision` and `addgroup` logic are still responsible for setup up this context. However this new context give us multiple benefits: * all writer use a same, unified, logic, * this context is programmatically enforced, * each write "session" as a clearly identified start and end. The post-pull sidedata update logic is still doing writing by end and will be adjusted in a later changesets. This change affect the concurrency checker test, because register the state of the file in the transaction sooner in `addrevision` (about as early as what `addgroup` would do), so the abort is rollbacking the other commit. I don't want to weaken the current main logic. Differential Revision: https://phab.mercurial-scm.org/D10605

revlog: preindent some code in _enforceinlinesize Indenting this beforehand will make a future changeset much simpler. Differential Revision: https://phab.mercurial-scm.org/D10604

revlog: preindent some code in addgroup Indenting this beforehand will make a future changeset much simpler. Differential Revision: https://phab.mercurial-scm.org/D10603

pyoxidizer: use Python 3.9 (BC) Newer versions of PyOxidizer use Python 3.9 by default. We previously pinned the version to 3.8 to facilitate porting to a new PyOxidizer version and diffing results. Now that the porting work is complete, let's bump Python to Python 3.9. This will effectively change our Windows Inno and WiX Python 3 installers from Python 3.8 to 3.9. .. bc:: Windows .msi and .exe installers now use Python 3.9 instead of Python 3.8. Differential Revision: https://phab.mercurial-scm.org/D10689

packaging: use PyOxidizer for producing WiX MSI installer We recently taught our in-tree PyOxidizer configuration file to produce MSI installers with WiX using PyOxidizer's built-in support for doing so. This commit changes our WiX + PyOxidizer installer generation code to use this functionality. After this change, all the Python packaging code is doing is the following: * Building HTML documentation * Making gettext available to the build process. * Munging CLI arguments to variables for the `pyoxidizer` execution. * Invoking `pyoxidizer build`. * Copying the produced `.msi` to the `dist/` directory. Applying this stack on stable and rebuilding the 5.8 MSI installer produced the following differences from the official 5.8 installer: * .exe and .pyd files aren't byte identical (this is expected). * Various .dist-info/ directories have different names due to older versions of PyOxidizer being buggy and not properly normalizing package names. (The new behavior is correct.) * Various *.dist-info/RECORD files are different due to content divergence of files (this is expected). * The python38.dll differs due to newer PyOxidizer shipping a newer version of Python 3.8. * We now ship python3.dll because PyOxidizer now includes this file by default. * The vcruntime140.dll differs because newer PyOxidizer installs a newer version. We also now ship a vcruntime140_1.dll because newer versions of the redistributable ship 2 files now. The WiX GUIDs and IDs of installed files have likely changed as a result of PyOxidizer's different mechanism for generating those identifiers. This means that an upgrade install of the MSI will replace files instead of doing an incremental update. This is likely harmless and we've incurred this kind of breakage before. As far as I can tell, the new PyOxidizer-built MSI is functionally equivalent to the old method. Once we drop support for Python 2.7 MSI installers, we can delete the WiX code from the repository. This commit temporarily drops support for extra `.wxs` files. We raise an exception instead of silently not using them, which I think is appropriate. We should be able to add support back in by injecting state into pyoxidizer.bzl via `--var`. I just didn't want to expend cognitive load to think about the solution as part of this series. Differential Revision: https://phab.mercurial-scm.org/D10688

packaging: extract invocation of pyoxidizer to own function I'll be refactoring how the WiX installer creation calls into pyoxidizer and will need a lower level function for facilitating that. The new `run_pyoxidizer()` builds our execution environment (with gettext available) and invokes `pyoxidizer`. Differential Revision: https://phab.mercurial-scm.org/D10687

packaging: rename run_pyoxidizer() I'm going to split this function up in a future commit and I'll want the name "run_pyoxidizer()" for a lower-level function for invoking `pyoxidizer`. Differential Revision: https://phab.mercurial-scm.org/D10686

packaging: move documentation HTML building to own function This is part of some light refactoring to enable us to use PyOxidizer for WiX MSI installer generation. Differential Revision: https://phab.mercurial-scm.org/D10685