cmdutil: avoid redefining write() function in export if possible Doing less work inside the loop just feels better to me.

cmdutil: extract closure that performs the actual export formatting This simplifies things a little by making the actual act of turning a revision into patch data a single function. After this, adding formatter support to `hg export` should be much simpler.

cmdutil: use a generator expression instead of a list comprehension

cmdutil: rename template param to export to fntemplate It's actually a template for the filename, not a formatter template.

cmdutil: comprehensively document the interface of export I want to make some improvements here, but in order to make future patches easier to review I want to document the current state of the world.

tests: hide warning from test-xdg.t The test-xdg.t test uses the system hgrc because it unsets HGRCPATH. If the system has an extension enabled that doesn't work with the development version of Mercurial it prints an 'extension failed to load' warning. Let's just hide that by piping stderr to /dev/null.

policy: define C module versions individually This allows us to bump the version for a single module without changing all of them.

repoview: rename '_getdynamicblockers' to 'revealedrevs' (API) Recent mailing list discussion made me realised we could clarify these. We make the function "public" to encourage extensions to wrap it and we use a more explicit name that mirror "hideablerevs".

repoview: move '_getdynamicblock' next to 'hideablerevs' There are the two functions that extensions should use to augment the hidding logic. It seem better to have them together at the top of the file.

python3: allow hgloader to work with lazy loaders Don't clobber the loader returned from find_spec. This brings `hg version` down from 0.27 seconds to 0.17.

init: turn on demandimport for Python 3.6 and above This uses the new demandimport implementation for Python 3 introduced in previous patches. This doesn't yet enhance performance because it isn't integrated with the custom source file loader we use on Python 3. We'll integrate the two in upcoming patches.

demandimport: add python 3 implementation This implementation uses the new importlib finder/loader functionality available in Python 3.5 and up. # no-check-commit

demandimport: move ignore list to __init__.py We're going to use the same ignore list for Python 3.

check: check modules in hgdemandimport A few places only check modules in mercurial and hgext. Add hgdemandimport to the list in those places.

demandimport: move to separate package In Python 3, demand loading is per-package. Keeping demandimport in the mercurial package would disable demand loading for any modules in mercurial.

import-checker: add a way to directly import certain symbols We'll use this for the 'demandimport' symbol in an upcoming patch.

check-code: allow skipping hasattr check in py3-only code hasattr is safe in Python 3, and in an upcoming patch we can't use util.safehasattr.

profiling: allow loading profiling extension before everything else 6d642ecf1a89 makes profiler start early without loading extensions. That makes it impossible for an extension to add customized profilers. This patch adds a special case: if a profiler is not found but an extension with the same name could be loaded, load that extension first, and expect it to have a "profile" contextmanager method. This allows customized profilers and extension setup time is still profiled.

extensions: allow loading a whitelisted subset of extensions This feature will be used by the next patch.

match: catch attempts to create case-insenstive exact matchers Exact matchers are only created internally (as opposed to from user input) based on a set of files that the caller collected before, so they should always match the list exactly (i.e. case-sensitively).

test: remove aliases forcing date, use default-date Now that we have the default-date by default and all code have been updated, remove the old commands alias that forced the date as they are not longer useful. Writing tests now should be easier for everyone now that all dates should be stable.

test: prepare tests for removing date aliases In the next patch, I'm gonna removing the global command aliases that force the epoch date but some tests either fail or their output change after that. Instead I'm copying the needed aliases in the test files that will otherwise change. Update test-rebase-obsolete.t because a revision hash is based on the epoch date after a 'commit --amend' and the output will change after removing date aliases. Update test-subrepo-git.t as the git subrepo doesn't use traditional date mechanisms. I'm not sure that updating the git subrepo to support default-date make sense. Add the commit alias to the test in order for making it pass after removing the date aliases globally.

devel: update blackbox to use default-date Blackbox now obeys the 'devel.default-date' option. As a side effect we can delete the mock for blackblox related tests.

devel: use default-date config field when creating obsmarkers Also use the default-date when creating obsmarkers. Currently they are created with the current date and without any option to force their value. To test the feature, we remove some of the many 'glob' used to match obsmarker date in the tests.

devel: activate default-date in tests It will make writing future tests more easier for everyone as we won't need to glob dates anymore in tests.

devel: add a config field to force dates to timestamp 0 Add a new config field named default-date under the devel section to force all implicits date to a specific value. If a explicit date is passed, it will override the default. This patch only affect changesets. Other usages (blackbox, obsmarkers) are updated in later patchs. The test runner is setting a bunch of alias to force the '--date' argument. We will replace theses aliases in a later patch.

ui: add the possiblity to get a date config field Add the method configdate to read a date from configuration. It uses the util.rawparsedate refactored earlier to support all standard date formats.

util: refactor util.parsedate to raises ValueError Split most of util.parsedate in util.rawparsedate and make it raises ValueError instead of error.Abort. The util.parsedate function is now just a shell function converting ValueError to error.Abort for existing users. I need to parse a date from config in a later patch and use util.rawparsedate with ui.configwith which expect a convert that raises ValueError.

match: implement __repr__() and update users (API) fsmonitor and debugignore currently access matcher fields that I would consider implementation details, namely patternspat, includepat, and excludepat. Let' instead implement __repr__() and have the few users use that instead. Marked (API) because the fields can now be None.

fsmonitor: fix silly "*kwargs" bug in merge.update() override Everyone knows that it's supposed to be spelled with two asterisks. It started failing in fad5e299cfc7 (update: accept --merge to allow merging across topo branches (issue5125), 2017-02-13) because until then there was only one argument that was covered by the kwargs, so *kwargs or **kwargs both worked (or at least that's what I think with my limited understanding of Python).

revset: make `hg log -r 'wdir()^'` work (issue4905) This patch catches the WdirUnsupported exception to support wdir()^.

revlog: raise WdirUnsupported when wdirrev is passed revlog.parentrevs() is called while evaluating ^ operator in revsets. When wdir is passed, it raises IndexError. This patch raises WdirUnsupported if wdir is passed in the function. The error will be caugth in future patches.

error: add a new exception named WdirUnsupported This series intents to support wdir() predicate with different operators like ~, ^. Currently the storage class don't support wdir(). This exception is introduced so that it can be raised from places where wdir() is passed and is not supported and we can catch this where we want to support the predicate. Also throwing an error at low level and catching at some higher level is better than using if-else especially for perf.

match: replace icasefsmatch() function by flag to regular match() match() will soon gain more logic and we don't want to duplicate that in icasefsmatch(), so merge the two functions instead and use a flag to get case-insensitive behavior.

match: delete icasefsmatcher now that it's same as matcher

match: pass in normalize() function to matchers This will let us delete icasefsmatcher.

match: don't print explicitly listed files with wrong case (BC) On case-insensitive file systems, if file A exists and you try to remove it (or add, etc.) by specifying a different case, you will see something like this: $ hg rm a removing file A I honestly found this surprising because it seems to me like it was explicitly listed by the user. Still, there is a comment in the code describing it, so it is very clearly intentional. The code was added in baa11dde8c0e (match: add a subclass for dirstate normalizing of the matched patterns, 2015-04-12). I'm going to do a lot of refactoring to matchers and the feature mentioned above is going to get in my way. I'm therefore removing it for the time being and we can hopefully add it back when I'm done.

match: move body of _normalize() to a static function matcher._normalize() no longer depends on any of the matcher's state, and making it static will enable further refactoring. Note that the subdirmatcher subclass calls _normalize(), so we can't remove it completely.

match: pass 'warn' argument to _normalize() for consistency No other arguments are passed via the matcher's state, so we should treat 'warn' the consistently. More importantly, this will let us make it a static function, which will help with further refactoring.

match: replace match class by match function (API) The matcher class is getting hard to understand. It will be easier to follow if we can break it up into simpler matchers that we then compose. I'm hoping to have one matcher that accepts regular (non-include) patterns, one for exact file matches, one that always matches (and maybe one that never does) and then compose them by intersection and difference. This patch takes a simple but important step towards that goal by making match.match() a function (and renaming the matcher class itself from "match" to "matcher"). The new function will eventually be responsible for creating the simple matchers and composing them. icasefsmatcher similarly gets a factory function (called "icasefsmatch"). I also moved the other factory functions nearby.

checklink: degrade gracefully on posix when fs is readonly (issue5511) In the unlucky case, checklink tries to make a new file for the symlink test to target. If the filesystem is readonly (perhaps due to permissions in a repo owned by someone else) we just report the filesystem as not supporting symlinks, since the user probably can't write anyway.

revlog: remove some revlogNG terminology RevlogNG is not such a good name when it is no longer the newest revlog version. Since we'll soon have revlog version 2, let's remove some references to it.

revlog: tweak wording and logic for flags validation First, the logic around the if..elif..elif was subtly wrong and sub-optimal because all branches would be tested as long as the revlog was valid. This patch changes things so it behaves like a switch statement over the revlog version. While I was here, I also tweaked error strings to make them consistent and to read better.

tests: tests for revlog version and flags loading We didn't have explicit test coverage of these branches. Better late than never.

perf: allow to clear the obsstore in 'perfvolatilesets' Loading the obsstore can become a large part of the time necessary to compute the important volatile set. We add a flag purging all known obsstore related data. For example, computing the 'bumped' set currently requires reading the full obsstore, so timing greatly differ with or without that flag: Without: ! bumped ! wall 0.005047 comb 0.000000 user 0.000000 sys 0.000000 (best of 446) With: ! bumped ! wall 0.512367 comb 0.510000 user 0.480000 sys 0.030000 (best of 15)

obsolete: test an important corner case Receiving markers affecting changeset we'll receives later is legitimate and not so uncommon case. Working on cache highlighted that this was only testing in the evolve extension. We add a test for this case in core.

largefiles: replace always() method, not _always field We will soon have matchers that don't have an _always field, so largefiles needs to stop assuming that they do. _always is only used by always(), so we safely replace that method instead.

merge with stable

cext: mark constant variables

cext: move util.h to cext tree Since util.h isn't useful in plain C module, it should be placed in CPython extension directory.

cext: move back finalization of dirstateTupleType where it should be

dispatch: setup color before pager for correct console information on windows Before this patch, "hg CMD --pager on" on Windows shows output unintentionally decorated with ANSI color escape sequences, if color mode is "auto". This issue occurs in steps below. 1. dispatch() invokes ui.pager() at detection of "--pager on" 2. stdout of hg process is redirected into stdin of pager process 3. "ui.formatted" = True, because isatty(stdout) is so before (2) 4. color module is loaded for colorization 5. color.w32effects = None, because GetConsoleScreenBufferInfo() fails on stdout redirected at (2) 6. "ansi" color mode is chosen, because of "not w32effects" 7. output is colorized in "ansi" mode because of "ui.formatted" = True Even if "ansi" color mode is chosen, ordinarily redirected stdout makes ui.formatted() return False, and colorization is avoided. But in this issue case, "ui.formatted" = True at (3) forces output to be colorized. For correct console information on win32, it is needed to ensure that color module is loaded before redirection of stdout for pagination. BTW, if any of enabled extensions has "colortable" attribute, this issue is avoided even before this patch, because color module is imported as a part of loading such extension, and extension loading occurs before setting up pager. For example, mq and keyword have "colortable".

cleanup: reuse existing wctx variables instead of calling repo[None] Incidentally, this apparently means we load .hgsub one time less as well, which affects a test case.

localrepo: extract bookmarkheads method to bookmarks.py This method is only used internally by destutil, and it's obscure enough I'm willing to just move it without a deprecation warning, especially since the new method has more constrained functionality. Design-wise I'd also like to get active bookmark handling folded into the bookmark store, so that we don't squirrel away an extra attribute for the active bookmark on the repository object.

tests: avoid renamed files triggering check-commit failures I had to flag some revisions to not undergo check-commit scrutiny today to get the tests to pass, because they renamed files that had existing failures which we're not cleaning up. Let's prevent that in the future by using the extended git diff format to elide moved code.

dispatch: make request accept additional reposetups chg needs special logic around repo object creation (like, collecting and reporting repo path to the master server). Adding "reposetup" to dispatch.request seems to be an easy and reasonably clean way to allow that.

cext: extract revlog/index parsing code to own C file parsers.c is ~3000 lines and ~2/3 of it is related to the revlog index type. We already have separate C source files for directory utilities and manifest parsing. I think the quite unwieldy revlog/index parsing code should be self-contained as well. I performed the extraction as a file copy then removed content from both sides in order to preserve file history and blame. As part of this, I also had to move the hexdigit table and function to a shared header since it is used by both parsers.c and revlog.c # no-check-commit

debugcommands: use temporary dict for its command table Instead, load the table by commands.py so the debug commands should always be populated. The table in debugcommands.py is unnamed so extension authors wouldn't be confused to wrap debugcommands.table in place of commands.table.

commands: move debugcommands and debugcomplete back to commands module These commands depend heavily on the commands table, so it doesn't make much sense to isolate them to debugcommands.py. This helps eliminating the future import cycle.

commands: move templates of common command options to cmdutil (API) The goal is to get rid of the debugcommands -> commands dependency. Since globalopts is the property of the commands, it's kept in the commands module.

import-checker: drop workaround for pure modules

policy: drop custom importer for pure modules

parsers: switch to policy importer # no-check-commit

mpatch: switch to policy importer

diffhelpers: switch to policy importer # no-check-commit

bdiff: switch to policy importer # no-check-commit

base85: switch to policy importer

osutil: switch to policy importer "make clean" is recommended to test this change, though C API compatibility should be preserved.

policy: add helper to import cext/pure module These functions are sysstr API since __import__() and getattr() hate byte strings on Python 3. There's a minor BC, which is ImportError will be raised if invalid HGMODULEPOLICY is specified. I think this is more desired behavior. We're planning to add strict checking for C API compatibility. This patch includes the stub for it.

demandimport: strictly check missing locals argument locals may be an empty dict.

localrepo: mark walk convenience method as deprecated (API)

debugcommands: use repo[None].walk instead of repo.walk

cmdutil: use repo[None].walk instead of repo.walk

largefiles: use repo[None].walk instead of repo.walk

parsers: add version to help detect breaking binary changes

osutil: add version to help detect breaking binary changes See the previous patch for why.

mpatch: add version to help detect breaking binary changes

diffhelpers: add version to help detect breaking binary changes

base85: add version to help detect breaking binary changes

bdiff: add version to help detect breaking binary changes Previously, we have no way to detect if a compiled .so file could be used or not, and blindly load it if it exists. Usually we carefully maintain compatibility of .so and fallback to pure code gracefully. But if we stick to the rules, certain nice changes will be impossible to make in a clean way. This patch adds a "version" constant to the module so we can detect inconsistency and take appropriate actions (warn, abort, fallback to pure, run make automatically) in module loader.

obsmarker: add an experimental flag controlling "operation" recording It seems better to introduce the experiment behind a flag for now as there are multiple concerns around the feature: * Storing operation increase the size of obsolescence markers significantly (+10-20%). * It performs poorly when exchanging markers (cannot combine command names, command name might be unknown remotely, etc)

run-tests: remove references to Python 2.6 These are the obvious ones. There is tons of code in this file implementing features from unittest that weren't present in Python 2.6. But that's for other patches.

dirstate: mark {begin,end}parentchange as deprecated (API)

merge: migrate to context manager for changing dirstate parents

localrepo: migrate to context manager for changing dirstate parents

context: migrate to context manager for changing dirstate parents

rebase: migrate to context manager for changing dirstate parents

mq: migrate to context manager for changing dirstate parents

dirstate: introduce new context manager for marking dirstate parent changes

contrib: make editmergeps able to work with notepad++ Notepad++ has a different FIRSTLINE argument, so needs special handling.

contrib: make editmergeps able to work with Windows GUI editors Using Start-Process -Wait makes it wait until the process finishes, which is necesssary for Windows GUI applications. My short testing also demonstrated that it does not hurt with command line vim.

extensions: show deprecation warning for the use of cmdutil.command Since this is a fundamental API for extensions, we set 1-year period until actually removing it.

extensions: prohibit registration of command without using @command (API) Detect the problem earlier for better error indication. I'm tired of teaching users that the mq extension is not guilty but the third-party extension is. https://bitbucket.org/tortoisehg/thg/issues?q=%27norepo%27

extensions: optionally print hint on import failure Test will be added by the next patch.

error: add hint to ProgrammingError As the hint isn't shown by the default exception handler, we need to print it manually. I've copied the "** " style from _exceptionwarning().

registrar: unindent superfluous "if True" block

registrar: switch @command decorator to class It overrides _funcregistrarbase._doregister() since the structure of the command table is quite different.

registrar: move cmdutil.command to registrar module (API) cmdutil.command wasn't a member of the registrar framework only for a historical reason. Let's make that happen. This patch keeps cmdutil.command as an alias for extension compatibility.

gendoc: make sure locale path is set before loading any modules Otherwise some messages wouldn't be translated depending on when the util was loaded.

fsmonitor: don't attempt state-leave if we didn't state-enter The state-enter command may not have been successful; for example, the watchman client session may have timed out if the user was busy/idle for a long period during a merge conflict resolution earlier in processing a rebase for a stack of diffs. It's cleaner (from the perspective of the watchman logs) to avoid issuing the state-leave command in these cases. Test Plan: ran `hg rebase --tool :merge -r '(draft() & date(-14)) - master::' -d master` and didn't observe any errors in the watchman logs or in the output from `watchman -p -j <<<'["subscribe", "/data/users/wez/fbsource", "wez", {"expression": ["name", ".hg/updatestate"]}]'`

fsmonitor: acquire localrepo.wlock prior to emitting hg.update state we see some weird things in the watchman logs where the mercurial process is seemingly confused about which hg.update state it is publishing through watchman. On closer examination, we're seeing conflicting pids for the clients involved and this implies a race. To resolve this, we extend the wlock around the state-enter/state-leave events that are emitted to watchman. Test Plan: Some manual testing: In one window, run this, and then checkout a different rev: ``` $ watchman -p -j <<<'["subscribe", "/data/users/wez/fbsource", "wez", {"expression": ["name", ".hg/updatestate"]}]' { "version": "4.9.0", "subscribe": "wez", "clock": "c:1495034090:814028:1:312576" } { "state-enter": "hg.update", "version": "4.9.0", "clock": "c:1495034090:814028:1:312596", "unilateral": true, "subscription": "wez", "metadata": { "status": "ok", "distance": 125, "rev": "a1275d79ffa6c58b53116c8ec401c275ca6c1e2a", "partial": false }, "root": "/data/users/wez/fbsource" } { "root": "/data/users/wez/fbsource", "metadata": { "status": "ok", "distance": 125, "rev": "a1275d79ffa6c58b53116c8ec401c275ca6c1e2a", "partial": false }, "subscription": "wez", "unilateral": true, "version": "4.9.0", "clock": "c:1495034090:814028:1:312627", "state-leave": "hg.update" } ``` Tailed the watchman log file and looked for invalid state assertion errors, then ran my `rebase-all` script to update/rebase all of my heads. Didn't trigger the error condition (but couldn't reliably trigger it previously anyway), and the output captured above shows that the states are being emitted correctly.

obsolete: move the 'isenabled' function at the top of the file That is a simple and important function so having it at the top next to the related constant seems better.

cache: make the cache updated callback easily accessible to extension This will help extension to benefit from this new logic. As a side effect this clarify the 'transaction' method a little bit.

obscheckhead: skip context creation to test phase On repository with many many heads, this is called often enough to show an overhead to appears in some profile. So we skip the 'changectx' creation to test the node phases.

tests: add missing 'test-push-checkheads-partial-C1.t' For some reason, this one test case did not made it with the others as they got upstreamed during the 4.2 cycle.

contrib: add editmerge version for powershell This just adds a translation of existing contrib/editmerge to powershell. It allows users on Windows to iteratively resolve conflicts in their editor of choice. # no-check-commit

run-tests: allow hg executable to be hg.exe When running tests on Windows (via msys), user sometimes does not want to run them against source hg, but against compiled hg.exe. For that purpose, --with-hg option can be used, but currently run-tests.py prints a warning if the value of this argument is not a file with basename 'hg'. This patch allows such file to be 'hg.exe'.

obsolete: add operation metadata to rebase/amend/histedit obsmarkers By recording what operation created the obsmarker, we can show very intuitive messages to the user in various UIs. For instance, log output could have messages like "Amended as XXX" to show why a commit is old and has an 'x' on it. @ ac28e3 durham / First commit | | o d4afe7 durham | | Second commit | | | x 8e9a5d (Amended as ac28e3) durham |/ First commit |

obsolete: invalidate "volatile" set cache after merging marker Adding markers to the repository might affect the set of obsolete changesets. So we most remove the "volatile" set who rely in that data. We add two missing invalidations after merging markers. This was caught by code change in the evolve extensions tests. This issues highlight that the current way to do things is a bit fragile, however we keep things simple for stable.

match: use match.prefix() in subdirmatcher It seems like the subdirmatcher should be checking if the matcher it's based on is matching prefixes. It was effectively doing that already because "prefix() == not always() and not anypats() and not isexact()", subdirmatcher was checking the first two parts of that condition and I don't think it will ever be given an "exact" matcher with it's directory name (because exact matchers are for matching files, not directories). Still, let's switch to using prefix() for clarity (and because I'm trying to remove code that reaches for matchers internals).

match: avoid accessing match._pathrestricted from subdirmatcher Accessing only the public API wherever possible helps us refactor matchers later.

match: override visitdir() the usual way in subdirmatcher Just override the function instead of replacing it on each instance.

match: make _fileroots a @propertycache and rename it to _fileset The files in the set are not necesserily roots of anything. Making it a @propertycache will help towards extracting a base class for matchers.

fsmonitor: use modern match API It seems like fsmonitor/__init__.py was based on a pretty old version of dirstate.py. Let's copy over the changes from the following two commits: 55c449345b10 (match: add isexact() method to hide internals, 2014-10-29) 3c346969c321 (dirstate: avoid match.files() in walk(), 2015-05-19)

diffstat: properly count lines starting in '--' or '++' (issue5479) Lines that start in '--' or '++' were previously not counted as deletions or additions in diffstat, resulting in incorrect addition/deletion counts. The bug was present if the start of the line, combined with the diff character resulted in '---' or '+++'. diffstatdata will now track, for each file, if it has moved pas the header section by looking for a line beginning with '@@'. Once that has happened, lines beginning with '-' or '+' will be counted for deletions and additions. Once a line beginning with 'diff' is found, the process starts over.

test-http-bad-server: match different in output Same as for previous update, the longer port number can affect this ouput.

config: make config.items() return a copy config.items() was iterating over a copy of the data for the the specified section on Python 2 by using .items(). However, on Python 3, items() does not make a copy, so let's switch to explicitly making a copy to make it safe on both Python 2 and Python 3.

filemerge: store error messages in module variables Copytracing may be disabled because it's too slow (see experimental.disablecopytrace config option). In that case user may get errors like 'local changed FILE which other deleted'. It would be nice to give user a hint to rerun command with `--config experimental.disablecopytrace=False`. To make it possible let's extract error message to variables so that extension may overwrite them.

run-tests: support multiple cases in .t test Sometimes we want to run similar tests with slightly different configurations. Previously we duplicate the test files. This patch introduces special "#testcases" syntax that allows a single .t file to contain multiple test cases. Defined cases could be tested using "#if". For example, if a test should behave the same with or without an experimental flag, we can add the following to the .t header: #testcases default experimental-a #if experimental-a $ cat >> $HGRCPATH << EOF > [experimental] > feature=a > EOF #endif The "experimental-a" block won't be executed when running the "default" test case.

revlog: rename constants (API) Feature flag constants don't need "NG" in the name because they will presumably apply to non-"NG" version revlogs. All feature flag constants should also share a similar naming convention to identify them as such. And, "RevlogNG" isn't a great internal name since it isn't obvious it maps to version 1 revlogs. Plus, "NG" (next generation) is only a good name as long as it is the latest version. Since we're talking about version 2, now is as good a time as any to move on from that naming.