# fix - rewrite file content in changesets and working copy

#

# Copyright 2018 Google LLC.

#

# This software may be used and distributed according to the terms of the

# GNU General Public License version 2 or any later version.

"""rewrite file content in changesets or working copy (EXPERIMENTAL)

Provides a command that runs configured tools on the contents of modified files,

writing back any fixes to the working copy or replacing changesets.

Here is an example configuration that causes :hg:`fix` to apply automatic

formatting fixes to modified lines in C++ code::

  [fix]

  clang-format:command=clang-format --assume-filename={rootpath}

  clang-format:linerange=--lines={first}:{last}

  clang-format:pattern=set:**.cpp or **.hpp

The :command suboption forms the first part of the shell command that will be

used to fix a file. The content of the file is passed on standard input, and the

fixed file content is expected on standard output. Any output on standard error

will be displayed as a warning. If the exit status is not zero, the file will

not be affected. A placeholder warning is displayed if there is a non-zero exit

status but no standard error output. Some values may be substituted into the

command::

  {rootpath}  The path of the file being fixed, relative to the repo root

  {basename}  The name of the file being fixed, without the directory path

If the :linerange suboption is set, the tool will only be run if there are

changed lines in a file. The value of this suboption is appended to the shell

command once for every range of changed lines in the file. Some values may be

substituted into the command::

  {first}   The 1-based line number of the first line in the modified range

  {last}    The 1-based line number of the last line in the modified range

Deleted sections of a file will be ignored by :linerange, because there is no

corresponding line range in the version being fixed.

By default, tools that set :linerange will only be executed if there is at least

one changed line range. This is meant to prevent accidents like running a code

formatter in such a way that it unexpectedly reformats the whole file. If such a

tool needs to operate on unchanged files, it should set the :skipclean suboption

to false.

The :pattern suboption determines which files will be passed through each

configured tool. See :hg:`help patterns` for possible values. If there are file

arguments to :hg:`fix`, the intersection of these patterns is used.

There is also a configurable limit for the maximum size of file that will be

processed by :hg:`fix`::

  [fix]

  maxfilesize = 2MB

Normally, execution of configured tools will continue after a failure (indicated

by a non-zero exit status). It can also be configured to abort after the first

such failure, so that no files will be affected if any tool fails. This abort

will also cause :hg:`fix` to exit with a non-zero status::

  [fix]

  failure = abort

When multiple tools are configured to affect a file, they execute in an order

defined by the :priority suboption. The priority suboption has a default value

of zero for each tool. Tools are executed in order of descending priority. The

execution order of tools with equal priority is unspecified. For example, you

could use the 'sort' and 'head' utilities to keep only the 10 smallest numbers

in a text file by ensuring that 'sort' runs before 'head'::

  [fix]

  sort:command = sort -n

  head:command = head -n 10

  sort:pattern = numbers.txt

  head:pattern = numbers.txt

  sort:priority = 2

  head:priority = 1

To account for changes made by each tool, the line numbers used for incremental

formatting are recomputed before executing the next tool. So, each tool may see

different values for the arguments added by the :linerange suboption.

Each fixer tool is allowed to return some metadata in addition to the fixed file

content. The metadata must be placed before the file content on stdout,

separated from the file content by a zero byte. The metadata is parsed as a JSON

value (so, it should be UTF-8 encoded and contain no zero bytes). A fixer tool

is expected to produce this metadata encoding if and only if the :metadata

suboption is true::

  [fix]

  tool:command = tool --prepend-json-metadata

  tool:metadata = true

The metadata values are passed to hooks, which can be used to print summaries or

perform other post-fixing work. The supported hooks are::

  "postfixfile"

    Run once for each file in each revision where any fixer tools made changes

    to the file content. Provides "$HG_REV" and "$HG_PATH" to identify the file,

    and "$HG_METADATA" with a map of fixer names to metadata values from fixer

    tools that affected the file. Fixer tools that didn't affect the file have a

    valueof None. Only fixer tools that executed are present in the metadata.

  "postfix"

    Run once after all files and revisions have been handled. Provides

    "$HG_REPLACEMENTS" with information about what revisions were created and

    made obsolete. Provides a boolean "$HG_WDIRWRITTEN" to indicate whether any

    files in the working copy were updated. Provides a list "$HG_METADATA"

    mapping fixer tool names to lists of metadata values returned from

    executions that modified a file. This aggregates the same metadata

    previously passed to the "postfixfile" hook.

Fixer tools are run the in repository's root directory. This allows them to read

configuration files from the working copy, or even write to the working copy.

The working copy is not updated to match the revision being fixed. In fact,

several revisions may be fixed in parallel. Writes to the working copy are not

amended into the revision being fixed; fixer tools should always write fixed

file content back to stdout as documented above.

"""

from __future__ import absolute_import

import collections

import itertools

import json

import os

import re

import subprocess

from mercurial.i18n import _

from mercurial.node import nullrev

from mercurial.node import wdirrev

from mercurial.utils import (

    procutil,

    stringutil,

)

from mercurial import (

    cmdutil,

    context,

    copies,

    error,

    mdiff,

    merge,

    obsolete,

    pycompat,

    registrar,

    scmutil,

    util,

    worker,

)

# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for

# extensions which SHIP WITH MERCURIAL. Non-mainline extensions should

# be specifying the version(s) of Mercurial they are tested with, or

# leave the attribute unspecified.

testedwith = 'ships-with-hg-core'

cmdtable = {}

command = registrar.command(cmdtable)

configtable = {}

configitem = registrar.configitem(configtable)

# Register the suboptions allowed for each configured fixer, and default values.

FIXER_ATTRS = {

    'command': None,

    'linerange': None,

    'pattern': None,

    'priority': 0,

    'metadata': 'false',

    'skipclean': 'true',

}

for key, default in FIXER_ATTRS.items():

    configitem('fix', '.*(:%s)?' % key, default=default, generic=True)

# A good default size allows most source code files to be fixed, but avoids

# letting fixer tools choke on huge inputs, which could be surprising to the

# user.

configitem('fix', 'maxfilesize', default='2MB')

# Allow fix commands to exit non-zero if an executed fixer tool exits non-zero.

# This helps users do shell scripts that stop when a fixer tool signals a

# problem.

configitem('fix', 'failure', default='continue')

def checktoolfailureaction(ui, message, hint=None):

    """Abort with 'message' if fix.failure=abort"""

    action = ui.config('fix', 'failure')

    if action not in ('continue', 'abort'):

        raise error.Abort(_('unknown fix.failure action: %s') % (action,),

                          hint=_('use "continue" or "abort"'))

    if action == 'abort':

        raise error.Abort(message, hint=hint)

allopt = ('', 'all', False, _('fix all non-public non-obsolete revisions'))

baseopt = ('', 'base', [], _('revisions to diff against (overrides automatic '

                             'selection, and applies to every revision being '

                             'fixed)'), _('REV'))

revopt = ('r', 'rev', [], _('revisions to fix'), _('REV'))

wdiropt = ('w', 'working-dir', False, _('fix the working directory'))

wholeopt = ('', 'whole', False, _('always fix every line of a file'))

usage = _('[OPTION]... [FILE]...')

@command('fix', [allopt, baseopt, revopt, wdiropt, wholeopt], usage,

        helpcategory=command.CATEGORY_FILE_CONTENTS)

def fix(ui, repo, *pats, **opts):

    """rewrite file content in changesets or working directory

    Runs any configured tools to fix the content of files. Only affects files

    with changes, unless file arguments are provided. Only affects changed lines

    of files, unless the --whole flag is used. Some tools may always affect the

    whole file regardless of --whole.

    If revisions are specified with --rev, those revisions will be checked, and

    they may be replaced with new revisions that have fixed file content.  It is

    desirable to specify all descendants of each specified revision, so that the

    fixes propagate to the descendants. If all descendants are fixed at the same

    time, no merging, rebasing, or evolution will be required.

    If --working-dir is used, files with uncommitted changes in the working copy

    will be fixed. If the checked-out revision is also fixed, the working

    directory will update to the replacement revision.

    When determining what lines of each file to fix at each revision, the whole

    set of revisions being fixed is considered, so that fixes to earlier

    revisions are not forgotten in later ones. The --base flag can be used to

    override this default behavior, though it is not usually desirable to do so.

    """

    opts = pycompat.byteskwargs(opts)

    if opts['all']:

        if opts['rev']:

            raise error.Abort(_('cannot specify both "--rev" and "--all"'))

        opts['rev'] = ['not public() and not obsolete()']

        opts['working_dir'] = True

    with repo.wlock(), repo.lock(), repo.transaction('fix'):

        revstofix = getrevstofix(ui, repo, opts)

        basectxs = getbasectxs(repo, opts, revstofix)

        workqueue, numitems = getworkqueue(ui, repo, pats, opts, revstofix,

                                           basectxs)

        fixers = getfixers(ui)

        # There are no data dependencies between the workers fixing each file

        # revision, so we can use all available parallelism.

        def getfixes(items):

            for rev, path in items:

                ctx = repo[rev]

                olddata = ctx[path].data()

                metadata, newdata = fixfile(ui, repo, opts, fixers, ctx, path,

                                            basectxs[rev])

                # Don't waste memory/time passing unchanged content back, but

                # produce one result per item either way.

                yield (rev, path, metadata,

                       newdata if newdata != olddata else None)

        results = worker.worker(ui, 1.0, getfixes, tuple(), workqueue,

                                threadsafe=False)

        # We have to hold on to the data for each successor revision in memory

        # until all its parents are committed. We ensure this by committing and

        # freeing memory for the revisions in some topological order. This

        # leaves a little bit of memory efficiency on the table, but also makes

        # the tests deterministic. It might also be considered a feature since

        # it makes the results more easily reproducible.

        filedata = collections.defaultdict(dict)

        aggregatemetadata = collections.defaultdict(list)

        replacements = {}

        wdirwritten = False

        commitorder = sorted(revstofix, reverse=True)

        with ui.makeprogress(topic=_('fixing'), unit=_('files'),

                             total=sum(numitems.values())) as progress:

            for rev, path, filerevmetadata, newdata in results:

                progress.increment(item=path)

                for fixername, fixermetadata in filerevmetadata.items():

                    aggregatemetadata[fixername].append(fixermetadata)

                if newdata is not None:

                    filedata[rev][path] = newdata

                    hookargs = {

                      'rev': rev,

                      'path': path,

                      'metadata': filerevmetadata,

                    }

                    repo.hook('postfixfile', throw=False,

                              **pycompat.strkwargs(hookargs))

                numitems[rev] -= 1

                # Apply the fixes for this and any other revisions that are

                # ready and sitting at the front of the queue. Using a loop here

                # prevents the queue from being blocked by the first revision to

                # be ready out of order.

                while commitorder and not numitems[commitorder[-1]]:

                    rev = commitorder.pop()

                    ctx = repo[rev]

                    if rev == wdirrev:

                        writeworkingdir(repo, ctx, filedata[rev], replacements)

                        wdirwritten = bool(filedata[rev])

                    else:

                        replacerev(ui, repo, ctx, filedata[rev], replacements)

                    del filedata[rev]

        cleanup(repo, replacements, wdirwritten)

        hookargs = {

            'replacements': replacements,

            'wdirwritten': wdirwritten,

            'metadata': aggregatemetadata,

        }

        repo.hook('postfix', throw=True, **pycompat.strkwargs(hookargs))

def cleanup(repo, replacements, wdirwritten):

    """Calls scmutil.cleanupnodes() with the given replacements.

    "replacements" is a dict from nodeid to nodeid, with one key and one value

    for every revision that was affected by fixing. This is slightly different

    from cleanupnodes().

    "wdirwritten" is a bool which tells whether the working copy was affected by

    fixing, since it has no entry in "replacements".

    Useful as a hook point for extending "hg fix" with output summarizing the

    effects of the command, though we choose not to output anything here.

    """

    replacements = {prec: [succ] for prec, succ in replacements.iteritems()}

    scmutil.cleanupnodes(repo, replacements, 'fix', fixphase=True)

def getworkqueue(ui, repo, pats, opts, revstofix, basectxs):

    """"Constructs the list of files to be fixed at specific revisions

    It is up to the caller how to consume the work items, and the only

    dependence between them is that replacement revisions must be committed in

    topological order. Each work item represents a file in the working copy or

    in some revision that should be fixed and written back to the working copy

    or into a replacement revision.

    Work items for the same revision are grouped together, so that a worker

    pool starting with the first N items in parallel is likely to finish the

    first revision's work before other revisions. This can allow us to write

    the result to disk and reduce memory footprint. At time of writing, the

    partition strategy in worker.py seems favorable to this. We also sort the

    items by ascending revision number to match the order in which we commit

    the fixes later.

    """

    workqueue = []

    numitems = collections.defaultdict(int)

    maxfilesize = ui.configbytes('fix', 'maxfilesize')

    for rev in sorted(revstofix):

        fixctx = repo[rev]

        match = scmutil.match(fixctx, pats, opts)

        for path in sorted(pathstofix(

                        ui, repo, pats, opts, match, basectxs[rev], fixctx)):

            fctx = fixctx[path]

            if fctx.islink():

                continue

            if fctx.size() > maxfilesize:

                ui.warn(_('ignoring file larger than %s: %s\n') %

                        (util.bytecount(maxfilesize), path))

                continue

            workqueue.append((rev, path))

            numitems[rev] += 1

    return workqueue, numitems

def getrevstofix(ui, repo, opts):

    """Returns the set of revision numbers that should be fixed"""

    revs = set(scmutil.revrange(repo, opts['rev']))

    for rev in revs:

        checkfixablectx(ui, repo, repo[rev])

    if revs:

        cmdutil.checkunfinished(repo)

        checknodescendants(repo, revs)

    if opts.get('working_dir'):

        revs.add(wdirrev)

        if list(merge.mergestate.read(repo).unresolved()):

            raise error.Abort('unresolved conflicts', hint="use 'hg resolve'")

    if not revs:

        raise error.Abort(

            'no changesets specified', hint='use --rev or --working-dir')

    return revs

def checknodescendants(repo, revs):

    if (not obsolete.isenabled(repo, obsolete.allowunstableopt) and

        repo.revs('(%ld::) - (%ld)', revs, revs)):

        raise error.Abort(_('can only fix a changeset together '

                            'with all its descendants'))

def checkfixablectx(ui, repo, ctx):

    """Aborts if the revision shouldn't be replaced with a fixed one."""

    if not ctx.mutable():

        raise error.Abort('can\'t fix immutable changeset %s' %

                          (scmutil.formatchangeid(ctx),))

    if ctx.obsolete():

        # It would be better to actually check if the revision has a successor.

        allowdivergence = ui.configbool('experimental',

                                        'evolution.allowdivergence')

        if not allowdivergence:

            raise error.Abort('fixing obsolete revision could cause divergence')

def pathstofix(ui, repo, pats, opts, match, basectxs, fixctx):

    """Returns the set of files that should be fixed in a context

    The result depends on the base contexts; we include any file that has

    changed relative to any of the base contexts. Base contexts should be

    ancestors of the context being fixed.

    """

    files = set()

    for basectx in basectxs:

        stat = basectx.status(fixctx, match=match, listclean=bool(pats),

                              listunknown=bool(pats))

        files.update(

            set(itertools.chain(stat.added, stat.modified, stat.clean,

                                stat.unknown)))

    return files