copies: avoid calling matcher if matcher.always() When storing copy information in the changesets (experimental.copies.read-from=changeset-only), this patch speeds up hg debugpathcopies FENNEC_58_0_2_BUILD1 FIREFOX_59_0b8_BUILD2 from 5.9s to 4.7s. At the start of this series (b162229e), that command took 18min. Differential Revision: https://phab.mercurial-scm.org/D6422

copies: avoid unnecessary copying of copy dict When storing copy information in the changesets, this patch speeds up hg debugpathcopies FENNEC_58_0_2_BUILD1 FIREFOX_59_0b8_BUILD2 from 11s to 5.9s. That command takes 6.2s when storing copy information in filelogs. Differential Revision: https://phab.mercurial-scm.org/D6421

copies: don't filter out copy targets created on other side of merge commit If file X is copied to Y on one side of merge and the other side creates Y (no copy), we would not mark that as copy. In the changeset-centric pathcopies() version, that was done by checking if the copy target existed on the other branch. Even though merge commits are pretty uncommon, it still turned out to be too expensive to load the manifest of the parents of merge commits. In a repo of mozilla-unified converted to storing copies in changesets, about 2m30s of `hg debugpathcopies FIREFOX_BETA_59_END FIREFOX_BETA_60_BASE` is spent on this check of merge commits. I tried to think of a way of storing more information in the changesets in order to cheaply detect these cases, but I couldn't think of a solution. So this patch simply removes those checks. For reference, these extra copies are reported from the aforementioned command after this patch: browser/base/content/sanitize.js -> browser/modules/Sanitizer.jsm testing/mozbase/mozprocess/tests/process_normal_finish_python.ini -> testing/mozbase/mozprocess/tests/process_normal_finish.ini testing/mozbase/mozprocess/tests/process_waittimeout_python.ini -> testing/mozbase/mozprocess/tests/process_waittimeout.ini testing/mozbase/mozprocess/tests/process_waittimeout_10s_python.ini -> testing/mozbase/mozprocess/tests/process_waittimeout_10s.ini Since these copies were created on one side of some merge, it still seems reasonable to include them, so I'm not even sure it's worse than filelog pathcopies(), just different. Differential Revision: https://phab.mercurial-scm.org/D6420

copies: do full filtering at end of _changesetforwardcopies() As mentioned earlier, pathcopies() is very slow when copies are stored in the changeset. Most of the cost comes from calling _chain() for every changeset, which is slow because it needs to read manifests. It needs to read manifests to be able to filter out copies that are were created in one commit and then deleted. (It also filters out copies that were created from a file that didn't exist in the starting revision, but that's a fixed revision across calls to _chain(), so it's much cheaper.) This patch changes from _chainandfilter() to just _chain() in the main loop in _changesetforwardcopies(). It instead removes copies that have subsequently been removed by using ctx.filesremoved(). We thus rely on that to be fast. It timed this command in mozilla-unified: hg debugpathcopies FIREFOX_59_0b3_BUILD2 FIREFOX_BETA_59_END It took 18s before and 1.1s after. It's still faster when copy information is stored in filelogs: 0.70s. It also still gets slow when there are merge commits involved, because we read manifests there too. We'll deal with that later. Differential Revision: https://phab.mercurial-scm.org/D6419

rust-filepatterns: add comment about Windows path handling As I replied to the Phabricator message, this is wrong. And I even suspect it wouldn't compile because of multiple type mismatches. I think, in Rust where type system is rock solid, we can live with UTF-8 strings except for the bottom storage layer and the top UI/command layer. We'll still have to get around undecodable characters not to be lost, but I think it's okay to drop such filenames from match result if they don't match in UTF-8 world, not in Latin-1 world.

rust-filepatterns: silence warning of non_upper_case_globals

rust: update Cargo.lock to include @generated comment cargo 1.34.0 of Debian sid inserts this comment, and I'm tired of reverting the change every time I do make local. https://github.com/rust-lang/cargo/commit/bd0e4a08471b8bc7957829b4fd294b8985d4fa2d

merge with stable

cborutil: fix streamencode() to handle subtypes Otherwise the template filter 'cbor' could crash because of bytes subclass: ValueError: do not know how to encode <class 'mercurial.encoding.safelocalstr'>

lfs: correct an error in the TODO file

cat: don't prefetch files unless the output requires it It's a waste to cache lfs blobs when cat'ing the raw data at best, but a hassle debugging when the blob is missing. I'm not sure if there are other commands that have '{data}' for output, and if there's a general way to prefetch on that keyword. It's interesting that the verbose output seems to leak into the JSON output, but that seems like an existing bug.

tracing: add support for emitting counters Differential Revision: https://phab.mercurial-scm.org/D6526

tracing: extract tracing-active logic to separate function I'm about to add support for counters, and want to avoid duplicating this logic. Differential Revision: https://phab.mercurial-scm.org/D6525

catapipe: add support for COUNTER events Differential Revision: https://phab.mercurial-scm.org/D6524

demandimport: add tracing coverage for Python 3 This makes things feel a little less mysterious when modules are being imported. Differential Revision: https://phab.mercurial-scm.org/D6523

export: don't prefetch *all* files in manifest `hg export` only shows changed files, not all files, but we still prefetched all files in cmdutil.export(). The same is true for the other commands calling cmdutil.exportfile(). That meant that `hg export` with remotefilelog (or lfs, I assume) could take much longer than expected because it would download all the files in the repo. Differential Revision: https://phab.mercurial-scm.org/D6532

remotefilelog: remove obsolete filtering of treemanifest directories I think this has been obsolete since 2cf18f46a1ce (narrow: only walk files within narrowspec also for committed revisions, 2018-09-28). Differential Revision: https://phab.mercurial-scm.org/D6531

py3: add test-dirstate-race2.t to list of passing tests This test was added new recently. The py3 buildbot found that it passes, so let's add it to the list of passing tests. Differential Revision: https://phab.mercurial-scm.org/D6530

strip: during merge allow strip only when -f is used This ensures to abort strip to `hg strip` when we have a merge in progress and allow it only when a `--force` flag is used. Differential Revision: https://phab.mercurial-scm.org/D6529

deltas: set estimated compression upper bound to "3x" instead of "10x" In pratice, we very rarely observer compression better than "3x" on manifest deltas. Having a more aggressive estimate significantly helps our pathological use case on a private repository. Here are a comparison of timings using different upper bound. Estimated compression | ø | ×10 | ×5 | ×3 | timing | 14.11 | 2.61 | 1.96 | 1.53 | We also tested the impact of this series on an array of public repositories. This shown no impact in either size nor timing. Full data set below for those interested. Size ---- Regarding size, not significant impact have been noticed on neither public nor private repositories. Here are the number we gathered on public repositories: zlib/upperbound | no | 10x | 5x | 3x mercurial | 5 875 730 | 5 875 730 | 5 875 730 | 5 875 730 pypy | 27 782 913 | 27 782 913 | 27 782 913 | 27 782 913 netbeans | 159 161 207 | 159 161 207 | 159 161 207 | 159 959 879 (+0.5%) mozilla-central | 323 841 642 | 323 841 642 | 323 841 642 | 319 867 519 (-2.5%) mozilla-try | 746 649 123 | 746 649 123 | 746 649 123 | 741 155 568 (-0.7%) private-repo | 1 485 287 294 | 1 485 287 294 | 1 485 287 294 | 1 409 248 382 (-5.1%) zstd/upperbound | no | 10x | 5x | 3x mercurial | 5 895 206 | 5 895 206 | 5 895 206 | 5 895 206 pypy | 28 689 230 | 28 689 230 | 28 689 230 | 28 689 230 netbeans | 157 636 387 | 157 636 387 | 157 636 387 | 159 692 678 (+1.3%) mozilla-central | 317 650 281 | 317 650 281 | 317 650 281 | 319 613 603 (+0.6%) mozilla-try | 737 555 275 | 737 555 275 | 737 555 275 | 738 079 473 (+0.1%) private-repo | 1 352 362 982 | 1 352 362 982 | 1 346 961 880 | 1 361 327 384 (+0.7%) Speed ------ Timing gathered using `hg perfrevlogwrite -m`. Value are in seconds. mercurial zlib | no | 10x | 5x | 3x | total | 65.551783 | 65.388887 | 65.260658 | 65.321199 | max | 0.034544 | 0.034571 | 0.034659 | 0.034521 | 99.99% | 0.034544 | 0.034571 | 0.034659 | 0.034521 | zstd | no | 10x | 5x | 3x | total | 49.118449 | 49.054062 | 48.753588 | 48.740230 | max | 0.009338 | 0.009239 | 0.009202 | 0.009178 | 99.99% | 0.007618 | 0.007639 | 0.007626 | 0.007621 | pypy zlib | no | 10x | 5x | 3x | total | 560.865984 | 558.983817 | 559.083815 | 559.349152 | max | 0.219614 | 0.215922 | 0.218112 | 0.218107 | 99.99% | 0.219614 | 0.215922 | 0.218112 | 0.218107 | zstd | no | 10x | 5x | 3x | total | 349.393280 | 347.395819 | 347.185407 | 345.643985 | max | 0.084143 | 0.083536 | 0.081834 | 0.082178 | 99.99% | 0.039445 | 0.039639 | 0.039612 | 0.039175 | netbeans zlib | no | 10x | 5x | 3x | total | 33103.327727 | 33314.932260 | 33211.745233 | 33345.891778 | max | 2.666852 | 2.672059 | 2.662453 | 2.662936 | 99.99% | 2.058772 | 2.070429 | 2.069569 | 2.064653 | zstd | no | 10x | 5x | 3x | total | 20112.102708 | 20095.879719 | 20083.390300 | 20123.221859 | max | 2.063482 | 2.062851 | 2.065229 | 2.060147 | 99.99% | 1.146647 | 1.143794 | 1.142933 | 1.146529 | mozilla zlib | no | 10x | 5x | 3x | total | 41374.102138 | 41418.816773 | 41381.956370 | 41334.280732 | max | 3.383474 | 3.387400 | 3.405711 | 3.387316 | 99.99% | 1.006755 | 1.005954 | 1.007700 | 1.007373 | zstd | no | 10x | 5x | 3x | total | 24689.691520 | 24643.939662 | 24664.630027 | 24664.512714 | max | 1.460822 | 1.449640 | 1.439747 | 1.465304 | 99.99% | 0.527111 | 0.527377 | 0.527807 | 0.527226 |

deltas: skip if projected compressed size is bigger than previous snapshot If we have a delta, we check constraints against a lower bound estimate of the resulting compressed delta. We then checks this projected size against the `size(snapshotⁿ) > size(snapshotⁿ⁺¹)` constraint. This allows to exclude potential base candidates before doing any expensive computation. This only apply to the intermediate-snapshot case since this constraint only apply to them. For some pathological cases of a private repository this step provide a further performance boost (timing from `hg perfrevlogwrite`): before: 3.010646 seconds after: 2.609307 seconds

deltas: skip if projected compressed size does not match text size constraint If we have a delta, we check constraints against a lower bound estimate of the resulting compressed delta. We then checks this projected size against the ½ⁿ size constraints. This allows to exclude potential base candidates before doing any expensive computation. This only apply to the intermediate-snapshot case since this constraint only apply to them. For some pathological cases of a private repository this step provide a further performance boost (timing from `hg perfrevlogwrite`): before: 3.145906 seconds after: 3.010646 seconds

deltas: accept and skip None return for delta info They are some extra computation that will shortcut the delta compression if the delta seems hopeless, returning None.

delta: move some delta chain related computation earlier in deltainfo They are some more optimization change that will make use of this in the function. So we retrieve the data earlier.

deltas: skip if projected delta size is bigger than previous snapshot Before computing any delta, we get a basic estimation of the delta size we can expect and the resulted compressed value. We then checks this projected size against the `size(snapshotⁿ) > size(snapshotⁿ⁺¹)` constraint. This allows to exclude potential base candidates before doing any expensive computation. This only apply to the intermediate-snapshot case since this constraint only apply to them. For some pathological cases of a private repository this step provide a significant performance boost (timing from `hg perfrevlogwrite`): before: 14.115908 seconds after: 3.145906 seconds

deltas: skip if projected delta size does not match text size constraint Before computing any delta, we get a basic estimation of the delta size we can expect and the resulted compressed value. We then checks this projected size against the ½ⁿ size constraints. This allows to exclude potential base candidates before doing any expensive computation. This only apply to the intermediate-snapshot case since this constraint only apply to them. In practice we only perform this new checks for the manifestlog. Manifest log combine two property: it is likely to have delta chain issue and its diffing/compression is fairly predictable. The initial author of this changeset is Valentin Gatien-Baron providing the initial idea and initial testing, Pierre-Yves David later consolidated the code in the right location and run more extensive testing.

revlog: add the option to track the expected compression upper bound There are various optimization we can do if we can estimate the size of delta before actually spending CPU compressing them. So we add a attributed dedicated to tracking that. We only use it on Manifest because (1) it structure is quite stable across all Mercurial repository so its compression ratio is fairly universal. This is the revlog with most extreme delta (cf the sparse-revlog optimization). This will be put to use in later changesets. Right now the compression upper bound is set to 10. This is a fairly conservative value (observed value is more around 3), but I prefer to be safe while introducing the optimization principles. We can tune the optimization threshold later.

perf: clarify some of the custom behavior of `perfrevlogwrite` This reduce the chance of developers being surprised by special cases.

perf: fix perfrevlogwrite --count documentation The help text was copy pasted from the previous option.

rust: switched to 'cargo rustc' in setup.py This is more flexible in the passing of additional flags, also what setuptools_rust does, giving less uncertainty about non-Linux platforms.