# fix - rewrite file content in changesets and working copy

#

# Copyright 2018 Google LLC.

#

# This software may be used and distributed according to the terms of the

# GNU General Public License version 2 or any later version.

"""rewrite file content in changesets or working copy (EXPERIMENTAL)

Provides a command that runs configured tools on the contents of modified files,

writing back any fixes to the working copy or replacing changesets.

Fixer tools are run in the repository's root directory. This allows them to read

configuration files from the working copy, or even write to the working copy.

The working copy is not updated to match the revision being fixed. In fact,

several revisions may be fixed in parallel. Writes to the working copy are not

amended into the revision being fixed; fixer tools MUST always read content to

be fixed from stdin, and write fixed file content back to stdout.

Here is an example configuration that causes :hg:`fix` to apply automatic

formatting fixes to modified lines in C++ code::

  [fix]

  clang-format:command=clang-format --assume-filename={rootpath}

  clang-format:linerange=--lines={first}:{last}

  clang-format:pattern=set:**.cpp or **.hpp

The :command suboption forms the first part of the shell command that will be

used to fix a file. The content of the file is passed on standard input, and the

fixed file content is expected on standard output. Any output on standard error

will be displayed as a warning. If the exit status is not zero, the file will

not be affected. A placeholder warning is displayed if there is a non-zero exit

status but no standard error output. Some values may be substituted into the

command::

  {rootpath}  The path of the file being fixed, relative to the repo root

  {basename}  The name of the file being fixed, without the directory path

If the :linerange suboption is set, the tool will only be run if there are

changed lines in a file. The value of this suboption is appended to the shell

command once for every range of changed lines in the file. Some values may be

substituted into the command::

  {first}   The 1-based line number of the first line in the modified range

  {last}    The 1-based line number of the last line in the modified range

Deleted sections of a file will be ignored by :linerange, because there is no

corresponding line range in the version being fixed.

By default, tools that set :linerange will only be executed if there is at least

one changed line range. This is meant to prevent accidents like running a code

formatter in such a way that it unexpectedly reformats the whole file. If such a

tool needs to operate on unchanged files, it should set the :skipclean suboption

to false.

The :pattern suboption determines which files will be passed through each

configured tool. See :hg:`help patterns` for possible values. However, all

patterns are relative to the repo root, even if that text says they are relative

to the current working directory. If there are file arguments to :hg:`fix`, the

intersection of these patterns is used.

There is also a configurable limit for the maximum size of file that will be

processed by :hg:`fix`::

  [fix]

  maxfilesize = 2MB

Normally, execution of configured tools will continue after a failure (indicated

by a non-zero exit status). It can also be configured to abort after the first

such failure, so that no files will be affected if any tool fails. This abort

will also cause :hg:`fix` to exit with a non-zero status::

  [fix]

  failure = abort

When multiple tools are configured to affect a file, they execute in an order

defined by the :priority suboption. The priority suboption has a default value

of zero for each tool. Tools are executed in order of descending priority. The

execution order of tools with equal priority is unspecified. For example, you

could use the 'sort' and 'head' utilities to keep only the 10 smallest numbers

in a text file by ensuring that 'sort' runs before 'head'::

  [fix]

  sort:command = sort -n

  head:command = head -n 10

  sort:pattern = numbers.txt

  head:pattern = numbers.txt

  sort:priority = 2

  head:priority = 1

To account for changes made by each tool, the line numbers used for incremental

formatting are recomputed before executing the next tool. So, each tool may see

different values for the arguments added by the :linerange suboption.

Each fixer tool is allowed to return some metadata in addition to the fixed file

content. The metadata must be placed before the file content on stdout,

separated from the file content by a zero byte. The metadata is parsed as a JSON

value (so, it should be UTF-8 encoded and contain no zero bytes). A fixer tool

is expected to produce this metadata encoding if and only if the :metadata

suboption is true::

  [fix]

  tool:command = tool --prepend-json-metadata

  tool:metadata = true

The metadata values are passed to hooks, which can be used to print summaries or

perform other post-fixing work. The supported hooks are::

  "postfixfile"

    Run once for each file in each revision where any fixer tools made changes

    to the file content. Provides "$HG_REV" and "$HG_PATH" to identify the file,

    and "$HG_METADATA" with a map of fixer names to metadata values from fixer

    tools that affected the file. Fixer tools that didn't affect the file have a

    value of None. Only fixer tools that executed are present in the metadata.

  "postfix"

    Run once after all files and revisions have been handled. Provides

    "$HG_REPLACEMENTS" with information about what revisions were created and

    made obsolete. Provides a boolean "$HG_WDIRWRITTEN" to indicate whether any

    files in the working copy were updated. Provides a list "$HG_METADATA"

    mapping fixer tool names to lists of metadata values returned from

    executions that modified a file. This aggregates the same metadata

    previously passed to the "postfixfile" hook.

"""

import collections

import itertools

import os

import re

import subprocess

from mercurial.i18n import _

from mercurial.node import (

    nullid,

    nullrev,

    wdirrev,

)

from mercurial.utils import procutil

from mercurial import (

    cmdutil,

    context,

    copies,

    error,

    logcmdutil,

    match as matchmod,

    mdiff,

    merge,

    mergestate as mergestatemod,

    pycompat,

    registrar,

    rewriteutil,

    scmutil,

    util,

    worker,

)

# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for

# extensions which SHIP WITH MERCURIAL. Non-mainline extensions should

# be specifying the version(s) of Mercurial they are tested with, or

# leave the attribute unspecified.

testedwith = b'ships-with-hg-core'

cmdtable = {}

command = registrar.command(cmdtable)

configtable = {}

configitem = registrar.configitem(configtable)

# Register the suboptions allowed for each configured fixer, and default values.

FIXER_ATTRS = {

    b'command': None,

    b'linerange': None,

    b'pattern': None,

    b'priority': 0,

    b'metadata': False,

    b'skipclean': True,

    b'enabled': True,

}

for key, default in FIXER_ATTRS.items():

    configitem(b'fix', b'.*:%s$' % key, default=default, generic=True)

# A good default size allows most source code files to be fixed, but avoids

# letting fixer tools choke on huge inputs, which could be surprising to the

# user.

configitem(b'fix', b'maxfilesize', default=b'2MB')

# Allow fix commands to exit non-zero if an executed fixer tool exits non-zero.

# This helps users do shell scripts that stop when a fixer tool signals a

# problem.

configitem(b'fix', b'failure', default=b'continue')

def checktoolfailureaction(ui, message, hint=None):

    """Abort with 'message' if fix.failure=abort"""

    action = ui.config(b'fix', b'failure')

    if action not in (b'continue', b'abort'):

        raise error.Abort(

            _(b'unknown fix.failure action: %s') % (action,),

            hint=_(b'use "continue" or "abort"'),

        )

    if action == b'abort':

        raise error.Abort(message, hint=hint)

allopt = (b'', b'all', False, _(b'fix all non-public non-obsolete revisions'))

baseopt = (

    b'',

    b'base',

    [],

    _(

        b'revisions to diff against (overrides automatic '

        b'selection, and applies to every revision being '

        b'fixed)'

    ),

    _(b'REV'),

)

revopt = (b'r', b'rev', [], _(b'revisions to fix (ADVANCED)'), _(b'REV'))

sourceopt = (

    b's',

    b'source',

    [],

    _(b'fix the specified revisions and their descendants'),

    _(b'REV'),

)

wdiropt = (b'w', b'working-dir', False, _(b'fix the working directory'))

wholeopt = (b'', b'whole', False, _(b'always fix every line of a file'))

usage = _(b'[OPTION]... [FILE]...')

@command(

    b'fix',

    [allopt, baseopt, revopt, sourceopt, wdiropt, wholeopt],

    usage,

    helpcategory=command.CATEGORY_FILE_CONTENTS,

)

def fix(ui, repo, *pats, **opts):

    """rewrite file content in changesets or working directory

    Runs any configured tools to fix the content of files. (See

    :hg:`help -e fix` for details about configuring tools.) Only affects files

    with changes, unless file arguments are provided. Only affects changed lines

    of files, unless the --whole flag is used. Some tools may always affect the

    whole file regardless of --whole.

    If --working-dir is used, files with uncommitted changes in the working copy

    will be fixed. Note that no backup are made.

    If revisions are specified with --source, those revisions and their

    descendants will be checked, and they may be replaced with new revisions

    that have fixed file content. By automatically including the descendants,

    no merging, rebasing, or evolution will be required. If an ancestor of the

    working copy is included, then the working copy itself will also be fixed,

    and the working copy will be updated to the fixed parent.

    When determining what lines of each file to fix at each revision, the whole

    set of revisions being fixed is considered, so that fixes to earlier

    revisions are not forgotten in later ones. The --base flag can be used to

    override this default behavior, though it is not usually desirable to do so.

    """

    opts = pycompat.byteskwargs(opts)

    cmdutil.check_at_most_one_arg(opts, b'all', b'source', b'rev')

    cmdutil.check_incompatible_arguments(

        opts, b'working_dir', [b'all', b'source']

    )

    with repo.wlock(), repo.lock(), repo.transaction(b'fix'):

        revstofix = getrevstofix(ui, repo, opts)

        basectxs = getbasectxs(repo, opts, revstofix)

        workqueue, numitems = getworkqueue(

            ui, repo, pats, opts, revstofix, basectxs

        )

        basepaths = getbasepaths(repo, opts, workqueue, basectxs)

        fixers = getfixers(ui)

        # Rather than letting each worker independently fetch the files

        # (which also would add complications for shared/keepalive

        # connections), prefetch them all first.

        _prefetchfiles(repo, workqueue, basepaths)

        # There are no data dependencies between the workers fixing each file

        # revision, so we can use all available parallelism.

        def getfixes(items):

            for srcrev, path, dstrevs in items:

                ctx = repo[srcrev]

                olddata = ctx[path].data()

                metadata, newdata = fixfile(

                    ui,

                    repo,

                    opts,

                    fixers,

                    ctx,

                    path,

                    basepaths,

                    basectxs[srcrev],

                )

                # We ungroup the work items now, because the code that consumes

                # these results has to handle each dstrev separately, and in

                # topological order. Because these are handled in topological

                # order, it's important that we pass around references to

                # "newdata" instead of copying it. Otherwise, we would be

                # keeping more copies of file content in memory at a time than

                # if we hadn't bothered to group/deduplicate the work items.

                data = newdata if newdata != olddata else None

                for dstrev in dstrevs:

                    yield (dstrev, path, metadata, data)

        results = worker.worker(

            ui, 1.0, getfixes, tuple(), workqueue, threadsafe=False

        )

        # We have to hold on to the data for each successor revision in memory

        # until all its parents are committed. We ensure this by committing and

        # freeing memory for the revisions in some topological order. This

        # leaves a little bit of memory efficiency on the table, but also makes

        # the tests deterministic. It might also be considered a feature since

        # it makes the results more easily reproducible.

        filedata = collections.defaultdict(dict)

        aggregatemetadata = collections.defaultdict(list)

        replacements = {}

        wdirwritten = False

        commitorder = sorted(revstofix, reverse=True)

        with ui.makeprogress(

            topic=_(b'fixing'), unit=_(b'files'), total=sum(numitems.values())

        ) as progress:

            for rev, path, filerevmetadata, newdata in results:

                progress.increment(item=path)

                for fixername, fixermetadata in filerevmetadata.items():

                    aggregatemetadata[fixername].append(fixermetadata)

                if newdata is not None:

                    filedata[rev][path] = newdata

                    hookargs = {

                        b'rev': rev,

                        b'path': path,

                        b'metadata': filerevmetadata,

                    }

                    repo.hook(

                        b'postfixfile',

                        throw=False,

                        **pycompat.strkwargs(hookargs)

                    )

                numitems[rev] -= 1

                # Apply the fixes for this and any other revisions that are

                # ready and sitting at the front of the queue. Using a loop here

                # prevents the queue from being blocked by the first revision to

                # be ready out of order.

                while commitorder and not numitems[commitorder[-1]]:

                    rev = commitorder.pop()

                    ctx = repo[rev]

                    if rev == wdirrev:

                        writeworkingdir(repo, ctx, filedata[rev], replacements)

                        wdirwritten = bool(filedata[rev])

                    else:

                        replacerev(ui, repo, ctx, filedata[rev], replacements)

                    del filedata[rev]

        cleanup(repo, replacements, wdirwritten)

        hookargs = {

            b'replacements': replacements,

            b'wdirwritten': wdirwritten,

            b'metadata': aggregatemetadata,

        }

        repo.hook(b'postfix', throw=True, **pycompat.strkwargs(hookargs))

def cleanup(repo, replacements, wdirwritten):

    """Calls scmutil.cleanupnodes() with the given replacements.

    "replacements" is a dict from nodeid to nodeid, with one key and one value

    for every revision that was affected by fixing. This is slightly different

    from cleanupnodes().

    "wdirwritten" is a bool which tells whether the working copy was affected by

    fixing, since it has no entry in "replacements".

    Useful as a hook point for extending "hg fix" with output summarizing the

    effects of the command, though we choose not to output anything here.

    """

    replacements = {prec: [succ] for prec, succ in replacements.items()}

    scmutil.cleanupnodes(repo, replacements, b'fix', fixphase=True)

def getworkqueue(ui, repo, pats, opts, revstofix, basectxs):

    """Constructs a list of files to fix and which revisions each fix applies to

    To avoid duplicating work, there is usually only one work item for each file

    revision that might need to be fixed. There can be multiple work items per

    file revision if the same file needs to be fixed in multiple changesets with

    different baserevs. Each work item also contains a list of changesets where

    the file's data should be replaced with the fixed data. The work items for

    earlier changesets come earlier in the work queue, to improve pipelining by

    allowing the first changeset to be replaced while fixes are still being

    computed for later changesets.

    Also returned is a map from changesets to the count of work items that might

    affect each changeset. This is used later to count when all of a changeset's

    work items have been finished, without having to inspect the remaining work

    queue in each worker subprocess.

    The example work item (1, "foo/bar.txt", (1, 2, 3)) means that the data of

    bar.txt should be read from revision 1, then fixed, and written back to

    revisions 1, 2 and 3. Revision 1 is called the "srcrev" and the list of

    revisions is called the "dstrevs". In practice the srcrev is always one of

    the dstrevs, and we make that choice when constructing the work item so that

    the choice can't be made inconsistently later on. The dstrevs should all

    have the same file revision for the given path, so the choice of srcrev is

    arbitrary. The wdirrev can be a dstrev and a srcrev.

    """

    dstrevmap = collections.defaultdict(list)

    numitems = collections.defaultdict(int)

    maxfilesize = ui.configbytes(b'fix', b'maxfilesize')

    for rev in sorted(revstofix):

        fixctx = repo[rev]

        match = scmutil.match(fixctx, pats, opts)

        for path in sorted(