revlog: no longer return the nodemap after parsing Now that both pure and cext version have a `nodemap` attribute, we do not need to return the `nodemap` object after parsing. This work is part of a refactoring to unify the revlog index and the nodemap. This unification prepare the use of a persistent nodemap. Differential Revision: https://phab.mercurial-scm.org/D7314

revlog: move the nodemap into the index object (for pure) This make the pure code closer to the C extension one. The ultimate goal is to merge the two into a single object and offer a unified API. This changeset focus on gathering the data on the same object. For now the code for `revlogoldindex` and `BaseIndexObject` index object are quite similar. However, there will be larger divergence later on, so I don't think is worth doing a base case. This work is part of a refactoring to unify the revlog index and the nodemap. This unification prepare the use of a persistent nodemap. Differential Revision: https://phab.mercurial-scm.org/D7313

revlog: introduce an explicit NodeMap class for pure code This class make the "pure" nodemap raise the same exception than the C-extension one. This is a step toward unifying nodemap and index, the class is not meant to survive on the long run. This work is part of a refactoring to unify the revlog index and the nodemap. This unification prepare the use of a persistent nodemap. There is a new `isinstance` call, it will be cleaned up in coming changesets. Differential Revision: https://phab.mercurial-scm.org/D7312

utils: move the `dirs` definition in pathutil (API) Before this change, the `dirs` class was accessible through the `mercurial.util` module. That module is expected to stay free of scm specific content. The `pathutil` destination has been selection by Martin von Zweigbergk. This work is part of a refactoring to unify the revlog index and the nodemap. This unification prepare the use of a persistent nodemap. Differential Revision: https://phab.mercurial-scm.org/D7311

packaging: stop installing i18n files The WiX installer has been shipping the content of the i18n/ directory since it was introduced in 2010 in 1e022c88a0a5. And the installer was subsequently refactored to only ship the .po files and hggettext. The .po files and the hggettext script are only used at build time to produce .mo files, which Mercurial does use at run-time. It doesn't make sense to install these files on Windows. So this commit stops doing that. This change further converges the file layout of the Inno and WiX installers. Differential Revision: https://phab.mercurial-scm.org/D7165

packaging: install hgk as hgk.tcl Giving it a file extension will give Windows a better chance at being able to execute it via standard means. This also makes WiX consistent with the Inno installer. Differential Revision: https://phab.mercurial-scm.org/D7164

packaging: add logo-droplets.svg The WiX installer adds this file. Let's add it to the Inno installer for consistency. Differential Revision: https://phab.mercurial-scm.org/D7163

packaging: install documentation to doc/ directory The source directory is "doc" and "doc" is used by the WiX installer. I think it makes sense to converge on the canonical path name. While we're here, we also lower the case because we can. (I think lower case names make more sense plus it is more consistent.) This only affects the Inno installer at the moment because it is the only installer using the staging code. Differential Revision: https://phab.mercurial-scm.org/D7162

packaging: always pass VERSION into Inno invocation The code in the Inno file was a holdover from before we had Python driving execution. With Python in the driver's seat, we can now have it resolve the version string and pass it into Inno, making the code easier to understand for people who aren't packaging gurus. Differential Revision: https://phab.mercurial-scm.org/D7161

packaging: remove hg.exe.local file <exe>.local files are used by Windows to set up DLL redirection. But these files are ignored if you embed an application manifest in your binary, which we do. So the existence of this file serves no purpose. So we remove it. Differential Revision: https://phab.mercurial-scm.org/D7160

packaging: stage installed files for Inno Previously, the Inno installer maintained its own mapping of source files to install location. (We have to maintain a similar mapping in the WiX installer.) Managing the explicit file layout for Windows packages is cumbersome and redundant. Every time you want to change the layout you need to change N locations. We frequently forget to do this and we only find out when people install Mercurial from our packages at release time. This commit starts the process of consolidating and simplifying the logic for managing the install layout on Windows. We introduce a list of install layout rules. These are simply source filenames (which can contain wildcards) and destination paths. The Inno packaging code has been updated to assemble all files into a staging directory that mirrors the final install layout. The list of files to add to the installer is derived by walking this staging directory and dynamically emitting the proper entries for the Inno Setup script. I diffed the file layout before and after this commit and there is no difference. Another benefit of this change is that it facilitates easier testing of the Windows install layout. Before, in order to test the final install layout, you needed to build an installer and run it. Now, you can stage files into the final layout and test from there, without running the installer. This should cut down on overhead when changing Windows code. Differential Revision: https://phab.mercurial-scm.org/D7159

packaging: process Inno Setup files with Jinja2 I want to make the Inno Setup files dynamically generated. This will enable us to do things like automatically derive the set of files to be packaged instead of having to manually keep lists of files in sync across installers. As the first step towards this, we process the Inno Setup files with Jinja2. As part of this conversion, we had to escape syntax in mercurial.iss that conflicts with Jinja2. I also took the opportunity to include modpath.iss via Jinja2 instead of using Inno's preprocessor. This keeps the Python code a bit simpler. Differential Revision: https://phab.mercurial-scm.org/D7158

packaging: install and run Inno files in a build directory Upcoming commits will refactor the Windows installers so their content is dynamically derived. In preparation for this, we copy the Inno files into a new build directory and run them from there. This required changing some relative paths to work from a new directory level but is otherwise a pretty straightforward change. Differential Revision: https://phab.mercurial-scm.org/D7157

packaging: consolidate CLI functionality into packaging.py Consolidating functionality for invoking code in the hgpackaging package through a single CLI entry point will make things simpler when we add more complexity to that package. For example, it will allow us to run things out of a virtualenv with third party packages. This commit consolidates functionality from the Inno and WiX build.py scripts into a new packaging.py script. That script simply creates a virtualenv and runs the CLI functionality in it. The new virtualenv is populated with jinja2 because I felt it easier to incorporate requirements file processing in this commit and we will soon use jinja2 in an upcoming commit. The unified CLI functionality will also make it easier to script other packaging workflows going forward. e.g. RPM, Debian, and macOS packaging. Differential Revision: https://phab.mercurial-scm.org/D7156

packaging: clean up excess packages for Inno This partially reverts e97ffa5511ba. A bunch of package dependencies were getting picked up because I ran pip-compile on Linux and the keyring package has some Linux-only dependencies. I have run pip-compile again - this time from Windows - to undo the addition of these dependencies. I also normalized whitespace to LF. Differential Revision: https://phab.mercurial-scm.org/D7155

unshelve: correct help string; unshelve does not accept file list The non-option arguments to `hg unshelve` are interpreted as the name of the shelve to unshelve, not the list of files to unshelve. While that functionality would probably be nice to have, that's well beyond the scope of this documentation fix. Differential Revision: https://phab.mercurial-scm.org/D7365

repoview: use class literal for creating filteredchangelog The type name is constant, so we don't need to create it dynamically using type(). As suggested by Yuya. Differential Revision: https://phab.mercurial-scm.org/D7364

vagrant: update vagrant image to buster from jessie I honestly have no idea if this works since I don't use vagrant, but looking around online suggests it should work fine. Differential Revision: https://phab.mercurial-scm.org/D7309

packaging: remove references to debian jessie Jessie is on life support at this point, so let's update the list of releases in the Makefiles. Differential Revision: https://phab.mercurial-scm.org/D7308

sqlitestore: remove superfluous r-prefixes on strings I did this file separately because it's a _ton_ of hits and it had some manual reformatting fallout as a result of black squeezing some expressions onto a single line. Differential Revision: https://phab.mercurial-scm.org/D7307

cleanup: remove pointless r-prefixes on single-quoted strings This is the promised second step on single-quoted strings. These had existed because our source transformer didn't turn r'' into b'', so we had tagged some strings as r-strings to get "native" strings on both Pythons. Now that the transformer is gone, we can dispense with this nonsense. Methodology: I ran hg locate 'set:added() or modified() or clean()' | egrep '.*\.py$' | xargs egrep --color=never -n -- \[\^b\]\[\^a-z\]r\'\[\^\'\\\\\]\*\'\[\^\'\ in an emacs grep-mode buffer, and then used a keyboard macro to iterate over the results and remove the r prefix as needed. # skip-blame removing unneeded r prefixes left over from Python 3 migration. Differential Revision: https://phab.mercurial-scm.org/D7306

rust-threads: force Rayon to respect the worker count in config As per the inline comment, this is a workaround because Rust code does not yet know how to read config files. Differential Revision: https://phab.mercurial-scm.org/D7310

phabricator: use context manager form of progress in uploadchunks Follow-up to 453079605242 / D7046. Differential Revision: https://phab.mercurial-scm.org/D7134

cleanup: remove pointless r-prefixes on double-quoted strings This is only double-quoted strings. I'll do single-quoted strings as a second step. These had existed because our source transformer didn't turn r"" into b"", so we had tagged some strings as r-strings to get "native" strings on both Pythons. Now that the transformer is gone, we can dispense with this nonsense. Methodology: I ran hg locate 'set:added() or modified() or clean()' | egrep '.*\.py$' | xargs egrep --color=never -n -- \[\^a-z\]r\"\[\^\"\\\\\]\*\"\[\^\"\] in an emacs grep-mode buffer, and then used a keyboard macro to iterate over the results and remove the r prefix as needed. # skip-blame removing unneeded r prefixes left over from Python 3 migration. Differential Revision: https://phab.mercurial-scm.org/D7305

formatting: remove another data-ogre from the config example The fix extension will pass the contents on stdin and read the output from stdout. By invoking `rustfmt` as `rustfmt {rootpath}`, we tell rustfmt to format the given path and update it on disk. That also makes rustftm not produce the formatted output to stdout. So the fix extension ends up writing out empty files. Differential Revision: https://phab.mercurial-scm.org/D7304

fix: replace str() by b'%d' for formatting integer Differential Revision: https://phab.mercurial-scm.org/D7303

zeroconf: improve the extension's documentation Differential Revision: https://phab.mercurial-scm.org/D7302

branchmap: pytype is confused about bytestr Differential Revision: https://phab.mercurial-scm.org/D7290

mail: suppress a pytype error that's just experimentally wrong It's probably more correct to pass a str here, but I'm not going to worry about it for now. Differential Revision: https://phab.mercurial-scm.org/D7283

dagparser: suppress some pytype errors around pycompat.bytestring I can't justify why we're getting these errors, but nothing I do fixes these handful of calls, so let's just move on with suppressions. Differential Revision: https://phab.mercurial-scm.org/D7277

encoding: add comment-based type hints for pytype Differential Revision: https://phab.mercurial-scm.org/D7275

merge with stable

dirs: reject consecutive slashes in paths We shouldn't ever see those, and the fuzzer go really excited that if it gives us a 65k string with 55k slashes in it we use a lot of RAM. This is a better fix than what I tried in D7105. It was suggested by Yuya, and I verified it does in fact cause the fuzzer to not OOM. This is a revision of D7234, but with the missing set of an error added. I added a unit test of the dirs behavior because I needed to reason more carefully about the failure modes around consecutive slashes. Differential Revision: https://phab.mercurial-scm.org/D7252

tests: quote $PYTHON in test-byteify-strings.t This is needed because the default install location is "Program Files" on Windows. Differential Revision: https://phab.mercurial-scm.org/D7251

repoview: define filteredchangelog as a top-level (non-local) class As suggested by Greg. This makes it easier for extensions to override the filtering. Differential Revision: https://phab.mercurial-scm.org/D7256

cmdutil: suppress bogus pytype errors pytype seems to handle our subclass of tuple (scmutil.status) especially poorly. We should probably file a bug. Differential Revision: https://phab.mercurial-scm.org/D7294

changegroup: suppress pytype error that's wrong Differential Revision: https://phab.mercurial-scm.org/D7293

changegroup: avoid shadowing a set with an int This confuses the heck out of pytype, and I basically agree with it that this shadowing is confusing. There's a chance this causes some memory to be freed later, but I think it's probably not worth worrying about for now. Differential Revision: https://phab.mercurial-scm.org/D7292

bundle2: more pytype suppressions around bytestring We clearly need to work around this somehow. Differential Revision: https://phab.mercurial-scm.org/D7291

archival: suppress some incorrect pytype failures Again, I suspect some missing coverage in type stubs for the stdlib. I'll revisit these later. Differential Revision: https://phab.mercurial-scm.org/D7288

simplemerge: disable a pytype error where it's just confused Differential Revision: https://phab.mercurial-scm.org/D7287

match: suppress error about subscripting an exception I get the sense I should file a bug about subscripting Exception subclasses. Differential Revision: https://phab.mercurial-scm.org/D7286

vfs: more attribute suppressions Differential Revision: https://phab.mercurial-scm.org/D7285

mdiff: mark diffopts as having dynamic attributes This class looks like we could move it to being an attrs and life would be better, but let's do that later. Differential Revision: https://phab.mercurial-scm.org/D7284

vfs: suppress some pytype errors around us using a private attribute Looking at threading._MainThread seems like we're probably a little unsupported, but since this code appears to work on both Python 2 and 3 I'm not going to sweat this for now. Differential Revision: https://phab.mercurial-scm.org/D7282

wireprototypes: disable pytype where it's just confused By inspection validnames is always a Set[bytes] here, but for some reason pytype is convinced it's a bytes. Let's disable the error for now. Differential Revision: https://phab.mercurial-scm.org/D7281

scmposix: another suppression on IOError subscripting As before, this will clear up when we move to Python 3-only. Differential Revision: https://phab.mercurial-scm.org/D7280

pvec: add an explicit type hint to help pytype Differential Revision: https://phab.mercurial-scm.org/D7279

procutil: suppress pytype warnings around windows-only attributes Differential Revision: https://phab.mercurial-scm.org/D7278

posix: add a pytype suppression This one seems a little sketchier than the others, but this will clean up when we're Python 3-only. Differential Revision: https://phab.mercurial-scm.org/D7276

color: suppress pytype warning on a windows-only module Differential Revision: https://phab.mercurial-scm.org/D7274

compression: tell pytype to not sweat a missing `zstd` module Differential Revision: https://phab.mercurial-scm.org/D7273

pytype: don't warn us about ignored-on-py3 metaclasses We can remove this when we're Python 3-only, but for now it's just too awkward to deal with and it's harmless. Differential Revision: https://phab.mercurial-scm.org/D7272

linelog: be more careful about types before looking for _target attribute Without this, pytype (correctly) complains that fakejge might not have a _cmprev attribute. We're operating on some constraints not expressible in the type system, so we detect the invalid case and raise a nicer exception now. Sadly, we also need an `assert` to give PyType a clue that we know what we're doing, but I can absolutely live with that. Differential Revision: https://phab.mercurial-scm.org/D7271

linelog: suppress annoying pytype warning about an ignored metaclass This isn't actively hurting us right now, and it's not trivial to set up this metaclass correctly on both Python 2 and Python 3 without dragging in all of `six`, but we still get lots of typechecking goodness with this error suppressed. Differential Revision: https://phab.mercurial-scm.org/D7270

hgignore: ignore generated pytype stubs Differential Revision: https://phab.mercurial-scm.org/D7269

hghave: add a check for pytype, Google's Python type checker I'm not unbiased, but I strongly prefer pytype to mypy: it does more inference, so we don't need as many annontations. In theory we can use both, but since I'm doing some tinkering with pytype I figured I'd just try setting it up. Differential Revision: https://phab.mercurial-scm.org/D7268

encoding: define local identify functions with explicit type comments This removes some obfuscation as far as pytype is concerned and corrects many bogus type errors. Differential Revision: https://phab.mercurial-scm.org/D7267

vfs: add NotImplementedError version of join Again, this helps out pytype. Differential Revision: https://phab.mercurial-scm.org/D7266

vfs: add a NotImplementedError implementation of __call__ This helps pytype considerably on this file. Differential Revision: https://phab.mercurial-scm.org/D7265

vfs: fix erroneous bytes constants Detected with pytype. Differential Revision: https://phab.mercurial-scm.org/D7264

pvec: migrate to modern integer division Detected with pytype. Differential Revision: https://phab.mercurial-scm.org/D7263

templateutil: fix a missing ABCMeta assignment Caught by pytype. Differential Revision: https://phab.mercurial-scm.org/D7262

pvec: fix overlooked chr() call Caught by pytype. Differential Revision: https://phab.mercurial-scm.org/D7261

vfs: another bytes-str confusion on thread name Differential Revision: https://phab.mercurial-scm.org/D7260

statprof: correctly always pass a str as the thread name Caught by pytype. Differential Revision: https://phab.mercurial-scm.org/D7259

py3: fix handling of ctrl keys in crecord (issue6213) The "keypressed" value in handlekeypressed() is a key name obtained by curses's getkey(); this can be a multibyte string for special keys like CTRL keys. Calling curses.unctrl() with such a value fails on Python 3 with a TypeError as described in issue6213. (On Python 2, this does not crash, but I'm not sure the result is correct, though it does no matter here.) So instead of calling unctrl(), we compare "keypressed" with the expected "^L" obtained by curses.ascii.ctrl("L").

py3: keep "keypressed" a native str in crecord This will help in the next changeset by avoiding a decode step. Also, the actual bytes conversion seems superfluous since values coming from curses's getkey() will be a native string. As a consequence, we open the "testcommands" file (used in test-interactive-curses.t) in text mode.

py3: compare response of crecord's confirmationwindow with str confirmationwindow() returns a native string, as a result of calling chr() on getch(). On Python 3, response.lower().startswith(b"y") leads to a TypeError. This fixes a crash when typing "r" in the curses interface of interactive commit.

py3: compare http server's command with a native string The "command" attribute is an str, so comparing with a bytes would not work on Python 3. This might solve issues in test-lfs-serve-access.t that happens sometimes (especially in CI): --- /hgwork/src/tests/test-lfs-serve-access.t +++ /hgwork/src/tests/test-lfs-serve-access.t.err @@ -163,11 +163,13 @@ $ cat $TESTTMP/access.log $TESTTMP/errors.log $LOCALIP - - [$LOGDATE$] "POST /missing/objects/batch HTTP/1.1" 404 - (glob) + $LOCALIP - - [05/Nov/2019 16:32:34] "{"objects": [{"oid": "f03217a32529a28a42d03b1244fe09b6e0f9fd06d7b966d4d50567be2abe6c0e", "size": 20}], "operation": "download"}" HTTPStatus.BAD_REQUEST - $LOCALIP - - [$LOGDATE$] "GET /subdir/mount/point?cmd=capabilities HTTP/1.1" 200 - (glob) $LOCALIP - - [$LOGDATE$] "GET /subdir/mount/point?cmd=batch HTTP/1.1" 200 - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (glob) $LOCALIP - - [$LOGDATE$] "GET /subdir/mount/point?cmd=getbundle HTTP/1.1" 200 - x-hgarg-1:bookmarks=1&bundlecaps=HG20%2Cbundle2%3DHG20%250Abookmarks%250Achangegroup%253D01%252C02%252C03%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Aphases%253Dheads%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps%250Arev-branch-cache%250Astream%253Dv2&cg=1&common=0000000000000000000000000000000000000000&heads=525251863cad618e55d483555f3d00a2ca99597e&listkeys=bookmarks&phases=1 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (glob) $LOCALIP - - [$LOGDATE$] "POST /subdir/mount/point/.git/info/lfs/objects/batch HTTP/1.1" 200 - (glob) $LOCALIP - - [$LOGDATE$] "GET /subdir/mount/point/.hg/lfs/objects/f03217a32529a28a42d03b1244fe09b6e0f9fd06d7b966d4d50567be2abe6c0e HTTP/1.1" 200 - (glob) + $LOCALIP - - [05/Nov/2019 16:32:34] code 400, message Bad request version ('"download"}') Blobs that already exist in the usercache are linked into the repo store, even though the client doesn't send the blob. @@ -195,6 +197,7 @@ server2/.hg/store/lfs/objects/f0/3217a32529a28a42d03b1244fe09b6e0f9fd06d7b966d4d50567be2abe6c0e $ "$PYTHON" $RUNTESTDIR/killdaemons.py $DAEMON_PIDS $ cat $TESTTMP/errors.log + $LOCALIP - - [05/Nov/2019 16:32:34] code 400, message Bad request version ('"download"}') $ cat >> $TESTTMP/lfsstoreerror.py <<EOF > import errno (from https://ci.hg.gregoryszorc.com/job-info/hg-committed-ca3dca416f8d5863ca6f5a4a6a6bb835dcd5feeb-debian10-cpython-3.7-0)

rust-status: remove dead code The `walk_explicit` function is only called when using a prefix matcher, which the Rust code does not yet support. This function will return in a future patch, probably with a different signature for performance reasons. With it, the `files` argument and its interface code can be removed for now. Differential Revision: https://phab.mercurial-scm.org/D7253

hghave: disallow symlinks on Windows Symlinks on Windows require either a special priviledge, or enabling Developer Mode. It's probably the latter that is enabled on the new CI machine. But since Mercurial itself is saying no to symlinks on Windows, the tests for symlinks shouldn't be attempted. This should fix a lot of the noise in the py3 tests. Differential Revision: https://phab.mercurial-scm.org/D7233

repoview: avoid wrapping changelog if there's nothing to filter This simplifies the code a bit by moving the optimizaton for no filtered revisions to one place. I assume it also makes working with repos without obsmarkers a little faster, but it doesn't seem significant. Differential Revision: https://phab.mercurial-scm.org/D7248

repoview: move changelog.flags() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7247

repoview: move changelog.parentrevs() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7246

repoview: move changelog.linkrev() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7245

repoview: move changelog.node() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7244

repoview: move changelog.rev() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7243

repoview: move changelog.strip() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7242

repoview: move changelog.headrevs() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7241

repoview: move changelog.revs() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7240

repoview: move changelog.__iter__() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7239

repoview: move changelog.__contains__() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7238

repoview: move changelog.tiprev() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7237

repoview: wrap changelog class when filtering The class doesn't yet do anything. I'll move the filter-aware overrides from the changelog class over one by one to this class in coming patches. That will leave the changelog class simpler and will centralize more of the filtering logic to repoview. I could not measure any performance difference. Differential Revision: https://phab.mercurial-scm.org/D7236

repoview: extract a function for wrapping changelog I would like to clean up the changelog class by moving out knowledge of filtering. The filtering will instead be done in a class that wraps the changelog. This patch prepares for that. Differential Revision: https://phab.mercurial-scm.org/D7235

revlog: move tiprev() from changelog up to revlog This makes the changelog override simpler and it seems more consistent. Differential Revision: https://phab.mercurial-scm.org/D7250

tests: fix a few `(no-windows !)` conditionals that are really `(symlink !)` The CI for py3 is assuming that symlinks are possible (they are when running as Administrator or when Developer Mode is enabled), and these popped up. Differential Revision: https://phab.mercurial-scm.org/D7232

rust-matchers: add `Matcher` trait and implement `AlwaysMatcher` In our quest of a faster Mercurial, we have arrived at the point where we need to implement the matchers in Rust. This RFC mainly for the `Matcher` trait to see if the changes proposed feel fine to people with more experience on the matter. While the `AlwaysMatcher` implementation is here as a trivial example, it should be the first step towards matchers use in Rust as it is currently the only supported one. Notable changes: - `exact` is renamed to `exact_match` - enums for `visit*` methods with `Recursive` instead of `'all'`, etc. - a new `roots`, separate from `file_set` - no `bad`, `explicitdir` or `traversedir` functions as they can be passed to the high functions instead of the matchers Thanks to Martin for suggesting the last two (most important) changes and for reaching out to help a few weeks ago. Differential Revision: https://phab.mercurial-scm.org/D7178

merge with stable

perf: add a way to benchmark `dirstate.status` Getting more details about time spend in this specific internal bit is meaningful.

largefiles: delete obsolete and unused repo.push() The function was removed from localrepo in 4d52e6eb98ea (locarepo: remove the `push` method (API), 2014-09-25). Differential Revision: https://phab.mercurial-scm.org/D7174

censor: document that some commands simply ignore censored data I can't see a benefit for hg grep to ever error out early when it encounters censored data.

grep: warn on censored revisions instead of erroring out We need most of the grep logic to go through in case we encounter a censored revision, so we just return a None body for a censored node, and we stop just short of trying to record matches with the contents of that censored body. The other parts such as recording that the censored file has been considered at this revision needs to go into the proper dicts. I have also gotten weary of all the abbreviations, so while I did a small refactor to move the file-data-getting operation into a common function, I also expanded the abbreviations of the relevant variables within this little function. Hopefully some day this helps someone figure out what all the abbreviations mean. Although the censoring docs currently state that some commands error out or are ignored depending on the `censor.policy` config, I cannot see a benefit for grep to ever stop dead in its tracks when a censored revision is encountered. I will also amend the docs to indicate that some commands, such as grep, unconditionally ignore censored revisions.

py3: enable legacy fs encoding to fix filename compatibility on Windows This patch is untested. I just followed the instruction: https://docs.python.org/3/whatsnew/3.6.html#pep-529-change-windows-filesystem-encoding-to-utf-8

rust-cpython: run cargo fmt

rust-cpython: remove useless PyResult<> from leak_immutable() The caller should know if the shared data is mutably borrowed or not.

rust-cpython: remove useless PyRefMut wrapper

rust-cpython: drop manual management of mutably_borrowed RefCell::borrow() should guarantee there's no mutable borrow. As a follow up, maybe PySharedState can be a pure data structure + function. Most ref-sharing business has already been moved to PySharedRef* and PyLeaked*.

rust-cpython: leverage RefCell::borrow() to guarantee there's no mutable ref Since the underlying value can't be mutably borrowed by PyLeaked, we don't have to manage yet another mutably-borrowed state. We can just rely on the RefCell implementation. Maybe we can add try_leak_immutable(), but this patch doesn't in order to keep the patch series not too long.

rust-cpython: remove useless Option<$leaked> from py_shared_iterator We no longer need to carefully drop the iterator when it's consumed. Mutation is allowed even if the iterator exists. There's a minor behavior change: next(iter) may return/raise something other than StopIteration if it's called after the iterator has been fully consumed, and if the Rust object isn't a FusedIterator.

rust-cpython: allow mutation unless leaked reference is borrowed In other words, mutation is allowed while a Python iterator holding PyLeaked exists. The iterator will be invalidated instead. We still need a borrow_count to prevent mutation while leaked data is dereferenced in Rust world, but most leak_count business is superseded by the generation counter. decrease_leak_count(py, true) will be removed soon.

rust-cpython: add generation counter to leaked reference This counter increments on borrow_mut() to invalidate existing leaked references. This is modeled after the iterator invalidation in Python. The other checks will be adjusted by the subsequent patches.

rust-cpython: add stub wrapper that'll prevent leaked data from being mutated In order to allow mutation of PySharedRefCell value while PyLeaked reference exists, we need yet another "borrow" scope where mutation is prohibited. try_borrow<'a> and try_borrow_mut<'a> defines the "borrow" scope <'a>. The subsequent patches will implement leak counter based on this scope. PyLeakedRef<T> and PyLeakedRefMut<T> could be unified to PyLeakedRef<&T> and PyLeakedRef<&mut T> respectively, but I didn't do that since it seemed a bit weird that deref_mut() would return a mutable reference to an immutable reference.

rust-cpython: rename PyLeakedRef to PyLeaked This series will make PyLeaked* behave more like a Python iterator, which means mutation of the owner object will be allowed and the leaked reference (i.e. the iterator) will be invalidated instead. I'll add PyLeakedRef/PyLeakedRefMut structs which will represent a "borrowed" state, and prevent the underlying value from being mutably borrowed while the leaked reference is in use: let shared = self.inner_shared(py); let leaked = shared.leak_immutable(); { let leaked_ref: PyLeakedRef<_> = leaked.borrow(py); shared.borrow_mut(); // panics since the underlying value is borrowed } shared.borrow_mut(); // allowed The relation between PyLeaked* structs is quite similar to RefCell/Ref/RefMut, but the implementation can't be reused because the borrowing state will have to be shared across objects having no lifetime relation. PyLeaked isn't named as PyLeakedCell since it isn't actually a cell in that leaked.borrow_mut() will require &mut self.

py3: don't use bytes with vars() or __dict__ Inspired by D7227. These were all the remaining instances I could find. Differential Revision: https://phab.mercurial-scm.org/D7230

Added signature for changeset ca3dca416f8d

Added tag 5.2 for changeset ca3dca416f8d

py3: add inline comment about encoding issue of str(Abort())

py3: do not reimplement Abort.__str__() on Python 2 It isn't necessary on Python 2, and the default implementation should be better than our BaseException_str() clone.

tests: write out file using bytes I/O The encoding of sys.stdout varies between Python versions. So using a one-liner to write a file from a Unicode string is not deterministic. This commit writes out the file using bytes I/O to ensure we have exactly the bytes we want in the file. This change fixes a test failure in Python 3.5/3.6. Differential Revision: https://phab.mercurial-scm.org/D7226

import-checker: open all source files as utf-8 Before, we opened in text mode and used the default encoding to interpret the bytes within. This caused problems interpreting some byte sequences in some files. This commit changes things to always open files as UTF-8, which makes the error go away. test-check-module-imports.t now passes on Python 3.5 and 3.6 with this change. Differential Revision: https://phab.mercurial-scm.org/D7225

localrepo: use str for lookup in vars() vars() returns a dict of str. So always use a native str for the key lookup. Differential Revision: https://phab.mercurial-scm.org/D7227

automation: install python3-venv Debian package Debian's python install has a crippled venv by default, as it is lacking ensurepip. When you try to run `python3 -m venv` it tells you to install `python3-venv`. So this commit does that in our automation environment so we can fully test installing Mercurial using venv+pip with the system Python. Differential Revision: https://phab.mercurial-scm.org/D7229

tests: look for ensurepip before using venv Debian appears to cripple the venv module by default by removing the associated ensurepip functionality. (The module isn't present at all.) This caused test-install.t to fail when using the Debian python3 unless the python3-venv package was installed. This commit introduces a new hghave requirement for detecting ensurepip and makes the Python 3 install variant conditional on its presence. This should make test-install.t pass when using an incomplete Debian Python. Differential Revision: https://phab.mercurial-scm.org/D7228

automation: avoid '~' in the temp directory on Windows If a long-ish username is used, the environment variable ends up with a '~' to be 8.3 path compatible. That in turn causes a handful of tests (mostly ssh related) to add quotes around $TESTMP. I have no AWS experience, so I have no idea if this is the proper way to do it. But I've hit this problem locally, and redirecting the directory is a workaround. I don't recall if the directory is created on demand by the test harness, but presumably if this is configured before the machine boots, Windows will do it for us. Differential Revision: https://phab.mercurial-scm.org/D7130

tests: use venv on Python 3 This test was failing in some Python 3 environments because `$PYTHON -m virtualenv` was somehow resulting in Python 2 being used. Why, I'm not sure. Python 3 includes virtualenv in the standard library as the `venv` module. This commit changes test-install.t to use `$PYTHON -m venv` on Python 3 and `$PYTHON -m virtualenv` on Python 2 (if available). I chose to make some test output duplicated because we can't have nested conditionals and there is no easy way to express ORing of hghave checks. Differential Revision: https://phab.mercurial-scm.org/D7224

tests: remove HGALLOWPYTHON3 reference This variable was removed from setup.py in c3e10f705a6c. Differential Revision: https://phab.mercurial-scm.org/D7223

run-tests: use byte strings for inserted output We were inserting str on Python 3 which resulted in mixed str/bytes types on the list. This would later blow up when trying to write str to the .err file opened in bytes mode. Differential Revision: https://phab.mercurial-scm.org/D7222

contrib: require Python 3.7 for byteify-strings.py bb509f39d387 made an error, it's actually 3.7 that introduced token.COMMENT. Differential Revision: https://phab.mercurial-scm.org/D7220