contrib/automation/hgautomation/cli.py
author Gregory Szorc <gregory.szorc@gmail.com>
Sat, 27 Apr 2019 11:48:26 -0700
changeset 42285 65b3ef162b39
parent 42283 d137a3d5ad41
child 42648 d80edcb0b30c
permissions -rw-r--r--
automation: initial support for running Linux tests Building on top of our Windows automation support, this commit implements support for performing automated tasks on remote Linux machines. Specifically, we implement support for running tests on ephemeral EC2 instances. This seems to be a worthwhile place to start, as building packages on Linux is more or less a solved problem because we already have facilities for building in Docker containers, which provide "good enough" reproducibility guarantees. The new `run-tests-linux` command works similarly to `run-tests-windows`: it ensures an AMI with hg dependencies is available, provisions a temporary EC2 instance with this AMI, pushes local changes to that instance via SSH, then invokes `run-tests.py`. Using this new command, I am able to run the entire test harness substantially faster then I am on my local machine courtesy of access to massive core EC2 instances: wall: 16:20 ./run-tests.py -l (i7-6700K) wall: 14:00 automation.py run-tests-linux --ec2-instance c5.2xlarge wall: 8:30 automation.py run-tests-linux --ec2-instance m5.4xlarge wall: 8:04 automation.py run-tests-linux --ec2-instance c5.4xlarge wall: 4:30 automation.py run-tests-linux --ec2-instance c5.9xlarge wall: 3:57 automation.py run-tests-linux --ec2-instance m5.12xlarge wall: 3:05 automation.py run-tests-linux --ec2-instance m5.24xlarge wall: 3:02 automation.py run-tests-linux --ec2-instance c5.18xlarge ~3 minute wall time to run pretty much the entire test harness is not too bad! The AMIs install multiple versions of Python. And the run-tests-linux command specifies which one to use: automation.py run-tests-linux --python system3 automation.py run-tests-linux --python 3.5 automation.py run-tests-linux --python pypy2.7 By default, the system Python 2.7 is used. Using this functionality, I was able to identity some unexpected test failures on PyPy! Included in the feature is support for running with alternate filesystems. You can simply pass --filesystem to the command to specify the type of filesystem to run tests on. When the ephemeral instance is started, a new filesystem will be created and tests will run from it: wall: 4:30 automation.py run-tests-linux --ec2-instance c5.9xlarge wall: 4:20 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem xfs wall: 4:24 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem tmpfs wall: 4:26 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem ext4 We also support multiple Linux distributions: $ automation.py run-tests-linux --distro debian9 total time: 298.1s; setup: 60.7s; tests: 237.5s; setup overhead: 20.4% $ automation.py run-tests-linux --distro ubuntu18.04 total time: 286.1s; setup: 61.3s; tests: 224.7s; setup overhead: 21.4% $ automation.py run-tests-linux --distro ubuntu18.10 total time: 278.5s; setup: 58.2s; tests: 220.3s; setup overhead: 20.9% $ automation.py run-tests-linux --distro ubuntu19.04 total time: 265.8s; setup: 42.5s; tests: 223.3s; setup overhead: 16.0% Debian and Ubuntu are supported because those are what I use and am most familiar with. It should be easy enough to add support for other distros. Unlike the Windows AMIs, Linux EC2 instances bill per second. So the cost to instantiating an ephemeral instance isn't as severe. That being said, there is some overhead, as it takes several dozen seconds for the instance to boot, push local changes, and build Mercurial. During this time, the instance is largely CPU idle and wasting money. Even with this inefficiency, running tests is relatively cheap: $0.15-$0.25 per full test run. A machine running tests as efficiently as these EC2 instances would cost say $6,000, so you can run the test harness a >20,000 times for the cost of an equivalent machine. Running tests in EC2 is almost certainly cheaper than buying a beefy machine for developers to use :) # no-check-commit because foo_bar function names Differential Revision: https://phab.mercurial-scm.org/D6319
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
42024
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     1
# cli.py - Command line interface for automation
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     2
#
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     3
# Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com>
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     4
#
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     5
# This software may be used and distributed according to the terms of the
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     6
# GNU General Public License version 2 or any later version.
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     7
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     8
# no-check-code because Python 3 native.
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     9
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    10
import argparse
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    11
import concurrent.futures as futures
42024
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    12
import os
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    13
import pathlib
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    14
import time
42024
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    15
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    16
from . import (
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    17
    aws,
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    18
    HGAutomation,
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    19
    linux,
42024
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    20
    windows,
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    21
)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    22
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    23
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    24
SOURCE_ROOT = pathlib.Path(os.path.abspath(__file__)).parent.parent.parent.parent
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    25
DIST_PATH = SOURCE_ROOT / 'dist'
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    26
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    27
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    28
def bootstrap_linux_dev(hga: HGAutomation, aws_region, distros=None,
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    29
                        parallel=False):
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    30
    c = hga.aws_connection(aws_region)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    31
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    32
    if distros:
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    33
        distros = distros.split(',')
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    34
    else:
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    35
        distros = sorted(linux.DISTROS)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    36
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    37
    # TODO There is a wonky interaction involving KeyboardInterrupt whereby
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    38
    # the context manager that is supposed to terminate the temporary EC2
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    39
    # instance doesn't run. Until we fix this, make parallel building opt-in
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    40
    # so we don't orphan instances.
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    41
    if parallel:
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    42
        fs = []
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    43
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    44
        with futures.ThreadPoolExecutor(len(distros)) as e:
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    45
            for distro in distros:
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    46
                fs.append(e.submit(aws.ensure_linux_dev_ami, c, distro=distro))
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    47
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    48
            for f in fs:
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    49
                f.result()
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    50
    else:
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    51
        for distro in distros:
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    52
            aws.ensure_linux_dev_ami(c, distro=distro)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    53
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
    54
42024
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    55
def bootstrap_windows_dev(hga: HGAutomation, aws_region):
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    56
    c = hga.aws_connection(aws_region)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    57
    image = aws.ensure_windows_dev_ami(c)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    58
    print('Windows development AMI available as %s' % image.id)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    59
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    60
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    61
def build_inno(hga: HGAutomation, aws_region, arch, revision, version):
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    62
    c = hga.aws_connection(aws_region)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    63
    image = aws.ensure_windows_dev_ami(c)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    64
    DIST_PATH.mkdir(exist_ok=True)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    65
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    66
    with aws.temporary_windows_dev_instances(c, image, 't3.medium') as insts:
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    67
        instance = insts[0]
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    68
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    69
        windows.synchronize_hg(SOURCE_ROOT, revision, instance)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    70
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    71
        for a in arch:
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    72
            windows.build_inno_installer(instance.winrm_client, a,
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    73
                                         DIST_PATH,
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    74
                                         version=version)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    75
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    76
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    77
def build_wix(hga: HGAutomation, aws_region, arch, revision, version):
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    78
    c = hga.aws_connection(aws_region)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    79
    image = aws.ensure_windows_dev_ami(c)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    80
    DIST_PATH.mkdir(exist_ok=True)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    81
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    82
    with aws.temporary_windows_dev_instances(c, image, 't3.medium') as insts:
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    83
        instance = insts[0]
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    84
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    85
        windows.synchronize_hg(SOURCE_ROOT, revision, instance)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    86
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    87
        for a in arch:
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    88
            windows.build_wix_installer(instance.winrm_client, a,
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    89
                                        DIST_PATH, version=version)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    90
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    91
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    92
def build_windows_wheel(hga: HGAutomation, aws_region, arch, revision):
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    93
    c = hga.aws_connection(aws_region)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    94
    image = aws.ensure_windows_dev_ami(c)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    95
    DIST_PATH.mkdir(exist_ok=True)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    96
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    97
    with aws.temporary_windows_dev_instances(c, image, 't3.medium') as insts:
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    98
        instance = insts[0]
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    99
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   100
        windows.synchronize_hg(SOURCE_ROOT, revision, instance)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   101
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   102
        for a in arch:
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   103
            windows.build_wheel(instance.winrm_client, a, DIST_PATH)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   104
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   105
42283
d137a3d5ad41 automation: add --version argument to build-all-windows-packages
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42277
diff changeset
   106
def build_all_windows_packages(hga: HGAutomation, aws_region, revision,
d137a3d5ad41 automation: add --version argument to build-all-windows-packages
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42277
diff changeset
   107
                               version):
42024
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   108
    c = hga.aws_connection(aws_region)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   109
    image = aws.ensure_windows_dev_ami(c)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   110
    DIST_PATH.mkdir(exist_ok=True)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   111
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   112
    with aws.temporary_windows_dev_instances(c, image, 't3.medium') as insts:
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   113
        instance = insts[0]
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   114
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   115
        winrm_client = instance.winrm_client
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   116
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   117
        windows.synchronize_hg(SOURCE_ROOT, revision, instance)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   118
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   119
        for arch in ('x86', 'x64'):
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   120
            windows.purge_hg(winrm_client)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   121
            windows.build_wheel(winrm_client, arch, DIST_PATH)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   122
            windows.purge_hg(winrm_client)
42283
d137a3d5ad41 automation: add --version argument to build-all-windows-packages
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42277
diff changeset
   123
            windows.build_inno_installer(winrm_client, arch, DIST_PATH,
d137a3d5ad41 automation: add --version argument to build-all-windows-packages
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42277
diff changeset
   124
                                         version=version)
42024
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   125
            windows.purge_hg(winrm_client)
42283
d137a3d5ad41 automation: add --version argument to build-all-windows-packages
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42277
diff changeset
   126
            windows.build_wix_installer(winrm_client, arch, DIST_PATH,
d137a3d5ad41 automation: add --version argument to build-all-windows-packages
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42277
diff changeset
   127
                                        version=version)
42024
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   128
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   129
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   130
def terminate_ec2_instances(hga: HGAutomation, aws_region):
42277
dd6a9723ae2b automation: don't create resources when deleting things
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42024
diff changeset
   131
    c = hga.aws_connection(aws_region, ensure_ec2_state=False)
42024
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   132
    aws.terminate_ec2_instances(c.ec2resource)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   133
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   134
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   135
def purge_ec2_resources(hga: HGAutomation, aws_region):
42277
dd6a9723ae2b automation: don't create resources when deleting things
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42024
diff changeset
   136
    c = hga.aws_connection(aws_region, ensure_ec2_state=False)
42024
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   137
    aws.remove_resources(c)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   138
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   139
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   140
def run_tests_linux(hga: HGAutomation, aws_region, instance_type,
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   141
                    python_version, test_flags, distro, filesystem):
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   142
    c = hga.aws_connection(aws_region)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   143
    image = aws.ensure_linux_dev_ami(c, distro=distro)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   144
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   145
    t_start = time.time()
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   146
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   147
    ensure_extra_volume = filesystem not in ('default', 'tmpfs')
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   148
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   149
    with aws.temporary_linux_dev_instances(
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   150
        c, image, instance_type,
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   151
        ensure_extra_volume=ensure_extra_volume) as insts:
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   152
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   153
        instance = insts[0]
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   154
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   155
        linux.prepare_exec_environment(instance.ssh_client,
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   156
                                       filesystem=filesystem)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   157
        linux.synchronize_hg(SOURCE_ROOT, instance, '.')
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   158
        t_prepared = time.time()
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   159
        linux.run_tests(instance.ssh_client, python_version,
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   160
                        test_flags)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   161
        t_done = time.time()
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   162
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   163
    t_setup = t_prepared - t_start
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   164
    t_all = t_done - t_start
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   165
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   166
    print(
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   167
        'total time: %.1fs; setup: %.1fs; tests: %.1fs; setup overhead: %.1f%%'
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   168
        % (t_all, t_setup, t_done - t_prepared, t_setup / t_all * 100.0))
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   169
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   170
42024
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   171
def run_tests_windows(hga: HGAutomation, aws_region, instance_type,
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   172
                      python_version, arch, test_flags):
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   173
    c = hga.aws_connection(aws_region)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   174
    image = aws.ensure_windows_dev_ami(c)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   175
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   176
    with aws.temporary_windows_dev_instances(c, image, instance_type,
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   177
                                             disable_antivirus=True) as insts:
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   178
        instance = insts[0]
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   179
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   180
        windows.synchronize_hg(SOURCE_ROOT, '.', instance)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   181
        windows.run_tests(instance.winrm_client, python_version, arch,
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   182
                          test_flags)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   183
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   184
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   185
def get_parser():
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   186
    parser = argparse.ArgumentParser()
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   187
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   188
    parser.add_argument(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   189
        '--state-path',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   190
        default='~/.hgautomation',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   191
        help='Path for local state files',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   192
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   193
    parser.add_argument(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   194
        '--aws-region',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   195
        help='AWS region to use',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   196
        default='us-west-1',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   197
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   198
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   199
    subparsers = parser.add_subparsers()
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   200
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   201
    sp = subparsers.add_parser(
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   202
        'bootstrap-linux-dev',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   203
        help='Bootstrap Linux development environments',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   204
    )
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   205
    sp.add_argument(
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   206
        '--distros',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   207
        help='Comma delimited list of distros to bootstrap',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   208
    )
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   209
    sp.add_argument(
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   210
        '--parallel',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   211
        action='store_true',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   212
        help='Generate AMIs in parallel (not CTRL-c safe)'
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   213
    )
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   214
    sp.set_defaults(func=bootstrap_linux_dev)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   215
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   216
    sp = subparsers.add_parser(
42024
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   217
        'bootstrap-windows-dev',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   218
        help='Bootstrap the Windows development environment',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   219
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   220
    sp.set_defaults(func=bootstrap_windows_dev)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   221
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   222
    sp = subparsers.add_parser(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   223
        'build-all-windows-packages',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   224
        help='Build all Windows packages',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   225
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   226
    sp.add_argument(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   227
        '--revision',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   228
        help='Mercurial revision to build',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   229
        default='.',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   230
    )
42283
d137a3d5ad41 automation: add --version argument to build-all-windows-packages
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42277
diff changeset
   231
    sp.add_argument(
d137a3d5ad41 automation: add --version argument to build-all-windows-packages
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42277
diff changeset
   232
        '--version',
d137a3d5ad41 automation: add --version argument to build-all-windows-packages
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42277
diff changeset
   233
        help='Mercurial version string to use',
d137a3d5ad41 automation: add --version argument to build-all-windows-packages
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42277
diff changeset
   234
    )
42024
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   235
    sp.set_defaults(func=build_all_windows_packages)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   236
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   237
    sp = subparsers.add_parser(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   238
        'build-inno',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   239
        help='Build Inno Setup installer(s)',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   240
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   241
    sp.add_argument(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   242
        '--arch',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   243
        help='Architecture to build for',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   244
        choices={'x86', 'x64'},
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   245
        nargs='*',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   246
        default=['x64'],
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   247
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   248
    sp.add_argument(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   249
        '--revision',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   250
        help='Mercurial revision to build',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   251
        default='.',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   252
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   253
    sp.add_argument(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   254
        '--version',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   255
        help='Mercurial version string to use in installer',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   256
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   257
    sp.set_defaults(func=build_inno)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   258
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   259
    sp = subparsers.add_parser(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   260
        'build-windows-wheel',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   261
        help='Build Windows wheel(s)',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   262
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   263
    sp.add_argument(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   264
        '--arch',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   265
        help='Architecture to build for',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   266
        choices={'x86', 'x64'},
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   267
        nargs='*',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   268
        default=['x64'],
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   269
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   270
    sp.add_argument(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   271
        '--revision',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   272
        help='Mercurial revision to build',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   273
        default='.',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   274
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   275
    sp.set_defaults(func=build_windows_wheel)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   276
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   277
    sp = subparsers.add_parser(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   278
        'build-wix',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   279
        help='Build WiX installer(s)'
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   280
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   281
    sp.add_argument(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   282
        '--arch',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   283
        help='Architecture to build for',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   284
        choices={'x86', 'x64'},
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   285
        nargs='*',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   286
        default=['x64'],
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   287
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   288
    sp.add_argument(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   289
        '--revision',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   290
        help='Mercurial revision to build',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   291
        default='.',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   292
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   293
    sp.add_argument(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   294
        '--version',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   295
        help='Mercurial version string to use in installer',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   296
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   297
    sp.set_defaults(func=build_wix)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   298
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   299
    sp = subparsers.add_parser(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   300
        'terminate-ec2-instances',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   301
        help='Terminate all active EC2 instances managed by us',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   302
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   303
    sp.set_defaults(func=terminate_ec2_instances)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   304
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   305
    sp = subparsers.add_parser(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   306
        'purge-ec2-resources',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   307
        help='Purge all EC2 resources managed by us',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   308
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   309
    sp.set_defaults(func=purge_ec2_resources)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   310
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   311
    sp = subparsers.add_parser(
42285
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   312
        'run-tests-linux',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   313
        help='Run tests on Linux',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   314
    )
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   315
    sp.add_argument(
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   316
        '--distro',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   317
        help='Linux distribution to run tests on',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   318
        choices=linux.DISTROS,
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   319
        default='debian9',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   320
    )
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   321
    sp.add_argument(
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   322
        '--filesystem',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   323
        help='Filesystem type to use',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   324
        choices={'btrfs', 'default', 'ext3', 'ext4', 'jfs', 'tmpfs', 'xfs'},
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   325
        default='default',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   326
    )
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   327
    sp.add_argument(
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   328
        '--instance-type',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   329
        help='EC2 instance type to use',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   330
        default='c5.9xlarge',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   331
    )
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   332
    sp.add_argument(
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   333
        '--python-version',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   334
        help='Python version to use',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   335
        choices={'system2', 'system3', '2.7', '3.5', '3.6', '3.7', '3.8',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   336
                 'pypy', 'pypy3.5', 'pypy3.6'},
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   337
        default='system2',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   338
    )
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   339
    sp.add_argument(
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   340
        'test_flags',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   341
        help='Extra command line flags to pass to run-tests.py',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   342
        nargs='*',
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   343
    )
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   344
    sp.set_defaults(func=run_tests_linux)
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   345
65b3ef162b39 automation: initial support for running Linux tests
Gregory Szorc <gregory.szorc@gmail.com>
parents: 42283
diff changeset
   346
    sp = subparsers.add_parser(
42024
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   347
        'run-tests-windows',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   348
        help='Run tests on Windows',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   349
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   350
    sp.add_argument(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   351
        '--instance-type',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   352
        help='EC2 instance type to use',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   353
        default='t3.medium',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   354
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   355
    sp.add_argument(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   356
        '--python-version',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   357
        help='Python version to use',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   358
        choices={'2.7', '3.5', '3.6', '3.7', '3.8'},
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   359
        default='2.7',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   360
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   361
    sp.add_argument(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   362
        '--arch',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   363
        help='Architecture to test',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   364
        choices={'x86', 'x64'},
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   365
        default='x64',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   366
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   367
    sp.add_argument(
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   368
        '--test-flags',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   369
        help='Extra command line flags to pass to run-tests.py',
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   370
    )
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   371
    sp.set_defaults(func=run_tests_windows)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   372
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   373
    return parser
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   374
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   375
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   376
def main():
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   377
    parser = get_parser()
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   378
    args = parser.parse_args()
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   379
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   380
    local_state_path = pathlib.Path(os.path.expanduser(args.state_path))
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   381
    automation = HGAutomation(local_state_path)
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   382
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   383
    if not hasattr(args, 'func'):
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   384
        parser.print_help()
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   385
        return
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   386
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   387
    kwargs = dict(vars(args))
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   388
    del kwargs['func']
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   389
    del kwargs['state_path']
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   390
b05a3e28cf24 automation: perform tasks on remote machines
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   391
    args.func(automation, **kwargs)