contrib/catapipe.py
author Joerg Sonnenberger <joerg@bec.de>
Fri, 30 Apr 2021 02:11:58 +0200
changeset 47043 12450fbea288
parent 43076 2372284d9457
child 48875 6000f5b25c9b
permissions -rwxr-xr-x
manifests: push down expected node length into the parser This strictly enforces the node length in the manifest lines according to what the repository expects. One test case moves large hash testing into the non-treemanifest part as treemanifests don't provide an interface for overriding just the node length for now. Differential Revision: https://phab.mercurial-scm.org/D10533
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
39252
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
     1
#!/usr/bin/env python3
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
     2
#
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
     3
# Copyright 2018 Google LLC.
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
     4
#
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
     5
# This software may be used and distributed according to the terms of the
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
     6
# GNU General Public License version 2 or any later version.
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
     7
"""Tool read primitive events from a pipe to produce a catapult trace.
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
     8
40491
c311424ea579 catapult: add a bit more documentation on how to use catapult tracing
Kyle Lippincott <spectral@google.com>
parents: 39935
diff changeset
     9
Usage:
c311424ea579 catapult: add a bit more documentation on how to use catapult tracing
Kyle Lippincott <spectral@google.com>
parents: 39935
diff changeset
    10
    Terminal 1: $ catapipe.py /tmp/mypipe /tmp/trace.json
c311424ea579 catapult: add a bit more documentation on how to use catapult tracing
Kyle Lippincott <spectral@google.com>
parents: 39935
diff changeset
    11
    Terminal 2: $ HGCATAPULTSERVERPIPE=/tmp/mypipe hg root
c311424ea579 catapult: add a bit more documentation on how to use catapult tracing
Kyle Lippincott <spectral@google.com>
parents: 39935
diff changeset
    12
    <ctrl-c catapipe.py in Terminal 1>
c311424ea579 catapult: add a bit more documentation on how to use catapult tracing
Kyle Lippincott <spectral@google.com>
parents: 39935
diff changeset
    13
    $ catapult/tracing/bin/trace2html /tmp/trace.json  # produce /tmp/trace.html
c311424ea579 catapult: add a bit more documentation on how to use catapult tracing
Kyle Lippincott <spectral@google.com>
parents: 39935
diff changeset
    14
    <open trace.html in your browser of choice; the WASD keys are very useful>
c311424ea579 catapult: add a bit more documentation on how to use catapult tracing
Kyle Lippincott <spectral@google.com>
parents: 39935
diff changeset
    15
    (catapult is located at https://github.com/catapult-project/catapult)
c311424ea579 catapult: add a bit more documentation on how to use catapult tracing
Kyle Lippincott <spectral@google.com>
parents: 39935
diff changeset
    16
39252
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    17
For now the event stream supports
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    18
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    19
  START $SESSIONID ...
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    20
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    21
and
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    22
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    23
  END $SESSIONID ...
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    24
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    25
events. Everything after the SESSIONID (which must not contain spaces)
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    26
is used as a label for the event. Events are timestamped as of when
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    27
they arrive in this process and are then used to produce catapult
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    28
traces that can be loaded in Chrome's about:tracing utility. It's
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    29
important that the event stream *into* this process stay simple,
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    30
because we have to emit it from the shell scripts produced by
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    31
run-tests.py.
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    32
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    33
Typically you'll want to place the path to the named pipe in the
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    34
HGCATAPULTSERVERPIPE environment variable, which both run-tests and hg
40491
c311424ea579 catapult: add a bit more documentation on how to use catapult tracing
Kyle Lippincott <spectral@google.com>
parents: 39935
diff changeset
    35
understand. To trace *only* run-tests, use HGTESTCATAPULTSERVERPIPE instead.
39252
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    36
"""
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    37
from __future__ import absolute_import, print_function
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    38
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    39
import argparse
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    40
import json
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    41
import os
39514
b29ec19748a7 contrib: use a monotonic timer in catapipe
Boris Feld <boris.feld@octobus.net>
parents: 39513
diff changeset
    42
import timeit
39252
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    43
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    44
_TYPEMAP = {
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    45
    'START': 'B',
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    46
    'END': 'E',
42475
ff562d711919 catapipe: add support for COUNTER events
Augie Fackler <augie@google.com>
parents: 40491
diff changeset
    47
    'COUNTER': 'C',
39252
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    48
}
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    49
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    50
_threadmap = {}
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    51
39514
b29ec19748a7 contrib: use a monotonic timer in catapipe
Boris Feld <boris.feld@octobus.net>
parents: 39513
diff changeset
    52
# Timeit already contains the whole logic about which timer to use based on
b29ec19748a7 contrib: use a monotonic timer in catapipe
Boris Feld <boris.feld@octobus.net>
parents: 39513
diff changeset
    53
# Python version and OS
b29ec19748a7 contrib: use a monotonic timer in catapipe
Boris Feld <boris.feld@octobus.net>
parents: 39513
diff changeset
    54
timer = timeit.default_timer
b29ec19748a7 contrib: use a monotonic timer in catapipe
Boris Feld <boris.feld@octobus.net>
parents: 39513
diff changeset
    55
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    56
39252
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    57
def main():
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    58
    parser = argparse.ArgumentParser()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    59
    parser.add_argument(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    60
        'pipe',
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    61
        type=str,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    62
        nargs=1,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    63
        help='Path of named pipe to create and listen on.',
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    64
    )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    65
    parser.add_argument(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    66
        'output',
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    67
        default='trace.json',
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    68
        type=str,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    69
        nargs='?',
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    70
        help='Path of json file to create where the traces ' 'will be stored.',
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    71
    )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    72
    parser.add_argument(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    73
        '--debug',
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    74
        default=False,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    75
        action='store_true',
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    76
        help='Print useful debug messages',
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
    77
    )
39252
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    78
    args = parser.parse_args()
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    79
    fn = args.pipe[0]
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    80
    os.mkfifo(fn)
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    81
    try:
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    82
        with open(fn) as f, open(args.output, 'w') as out:
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    83
            out.write('[\n')
39514
b29ec19748a7 contrib: use a monotonic timer in catapipe
Boris Feld <boris.feld@octobus.net>
parents: 39513
diff changeset
    84
            start = timer()
39252
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    85
            while True:
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    86
                ev = f.readline().strip()
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    87
                if not ev:
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    88
                    continue
39514
b29ec19748a7 contrib: use a monotonic timer in catapipe
Boris Feld <boris.feld@octobus.net>
parents: 39513
diff changeset
    89
                now = timer()
39252
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    90
                if args.debug:
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    91
                    print(ev)
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    92
                verb, session, label = ev.split(' ', 2)
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    93
                if session not in _threadmap:
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
    94
                    _threadmap[session] = len(_threadmap)
42475
ff562d711919 catapipe: add support for COUNTER events
Augie Fackler <augie@google.com>
parents: 40491
diff changeset
    95
                if verb == 'COUNTER':
ff562d711919 catapipe: add support for COUNTER events
Augie Fackler <augie@google.com>
parents: 40491
diff changeset
    96
                    amount, label = label.split(' ', 1)
ff562d711919 catapipe: add support for COUNTER events
Augie Fackler <augie@google.com>
parents: 40491
diff changeset
    97
                    payload_args = {'value': int(amount)}
ff562d711919 catapipe: add support for COUNTER events
Augie Fackler <augie@google.com>
parents: 40491
diff changeset
    98
                else:
ff562d711919 catapipe: add support for COUNTER events
Augie Fackler <augie@google.com>
parents: 40491
diff changeset
    99
                    payload_args = {}
39252
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
   100
                pid = _threadmap[session]
39935
cb9b23ec093c catapipe: fix bug introduced when switching to the timer api
Augie Fackler <augie@google.com>
parents: 39514
diff changeset
   101
                ts_micros = (now - start) * 1000000
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
   102
                out.write(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
   103
                    json.dumps(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
   104
                        {
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
   105
                            "name": label,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
   106
                            "cat": "misc",
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
   107
                            "ph": _TYPEMAP[verb],
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
   108
                            "ts": ts_micros,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
   109
                            "pid": pid,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
   110
                            "tid": 1,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
   111
                            "args": payload_args,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
   112
                        }
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
   113
                    )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
   114
                )
39252
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
   115
                out.write(',\n')
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
   116
    finally:
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
   117
        os.unlink(fn)
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
   118
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42475
diff changeset
   119
39252
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
   120
if __name__ == '__main__':
9a81f126f9fa contrib: new script to read events from a named pipe and emit catapult traces
Augie Fackler <augie@google.com>
parents:
diff changeset
   121
    main()