tests/f
author Arseniy Alekseyev <aalekseyev@janestreet.com>
Fri, 26 Apr 2024 19:10:35 +0100
changeset 51626 865efc020c33
parent 49874 e9d06ed64161
permissions -rwxr-xr-x
dirstate: remove the python-side whitelist of allowed matchers This whitelist is too permissive because it allows matchers that contain disallowed ones deep inside, for example through `intersectionmatcher`. It is also too restrictive because it doesn't pass through some of the matchers we support, such as `patternmatcher`. It's also unnecessary because unsupported matchers raise `FallbackError` and we fall back anyway. Making this change makes more of the tests use rust code path, and therefore subtly change behavior. For example, rust status in largefiles repos seems to have strange behavior.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
45830
c102b704edb5 global: use python3 in shebangs
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43691
diff changeset
     1
#!/usr/bin/env python3
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
     2
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
     3
"""
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
     4
Utility for inspecting files in various ways.
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
     5
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
     6
This tool is like the collection of tools found in a unix environment but are
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
     7
cross platform and stable and suitable for our needs in the test suite.
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
     8
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
     9
This can be used instead of tools like:
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    10
  [
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    11
  dd
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    12
  find
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    13
  head
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    14
  hexdump
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    15
  ls
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    16
  md5sum
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    17
  readlink
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    18
  sha1sum
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    19
  stat
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    20
  tail
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    21
  test
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    22
  readlink.py
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    23
  md5sum.py
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    24
"""
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    25
29160
0362605b82cf py3: make tests/f use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents: 28046
diff changeset
    26
36261
c69e78ef2b54 py3: port f to Python 3
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35472
diff changeset
    27
import binascii
29160
0362605b82cf py3: make tests/f use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents: 28046
diff changeset
    28
import glob
29233
318534bb5dfd tests: make 'f' utility import hashlib unconditionally
Yuya Nishihara <yuya@tcha.org>
parents: 29230
diff changeset
    29
import hashlib
29160
0362605b82cf py3: make tests/f use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents: 28046
diff changeset
    30
import optparse
0362605b82cf py3: make tests/f use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents: 28046
diff changeset
    31
import os
0362605b82cf py3: make tests/f use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents: 28046
diff changeset
    32
import re
0362605b82cf py3: make tests/f use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents: 28046
diff changeset
    33
import sys
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    34
43691
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
    35
49874
e9d06ed64161 tests: drop py2 support from `f` utility
Matt Harbison <matt_harbison@yahoo.com>
parents: 49556
diff changeset
    36
def iterbytes(s):
e9d06ed64161 tests: drop py2 support from `f` utility
Matt Harbison <matt_harbison@yahoo.com>
parents: 49556
diff changeset
    37
    for i in range(len(s)):
e9d06ed64161 tests: drop py2 support from `f` utility
Matt Harbison <matt_harbison@yahoo.com>
parents: 49556
diff changeset
    38
        yield s[i : i + 1]
34270
3db2365d43e4 tests: update `f` helper script to work on Python 3
Augie Fackler <augie@google.com>
parents: 32898
diff changeset
    39
43691
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
    40
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    41
def visit(opts, filenames, outfile):
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    42
    """Process filenames in the way specified in opts, writing output to
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    43
    outfile."""
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    44
    for f in sorted(filenames):
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    45
        isstdin = f == '-'
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    46
        if not isstdin and not os.path.lexists(f):
34270
3db2365d43e4 tests: update `f` helper script to work on Python 3
Augie Fackler <augie@google.com>
parents: 32898
diff changeset
    47
            outfile.write(b'%s: file not found\n' % f.encode('utf-8'))
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    48
            continue
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    49
        quiet = opts.quiet and not opts.recurse or isstdin
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    50
        isdir = os.path.isdir(f)
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    51
        islink = os.path.islink(f)
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    52
        isfile = os.path.isfile(f) and not islink
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    53
        dirfiles = None
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    54
        content = None
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    55
        facts = []
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    56
        if isfile:
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    57
            if opts.type:
36261
c69e78ef2b54 py3: port f to Python 3
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35472
diff changeset
    58
                facts.append(b'file')
49556
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
    59
            needs_reading = (
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
    60
                opts.hexdump,
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
    61
                opts.dump,
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
    62
                opts.md5,
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
    63
                opts.sha1,
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
    64
                opts.raw_sha1,
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
    65
                opts.sha256,
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
    66
            )
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
    67
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
    68
            if any(needs_reading):
36948
0585337ea787 cleanup: fix some latent open(path).read() et al calls we previously missed
Augie Fackler <augie@google.com>
parents: 36261
diff changeset
    69
                with open(f, 'rb') as fobj:
0585337ea787 cleanup: fix some latent open(path).read() et al calls we previously missed
Augie Fackler <augie@google.com>
parents: 36261
diff changeset
    70
                    content = fobj.read()
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    71
        elif islink:
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    72
            if opts.type:
36261
c69e78ef2b54 py3: port f to Python 3
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35472
diff changeset
    73
                facts.append(b'link')
40246
be0a5d2d5c78 tests: fix last failure in test-tools.t
Augie Fackler <augie@google.com>
parents: 38364
diff changeset
    74
            content = os.readlink(f).encode('utf8')
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    75
        elif isstdin:
34270
3db2365d43e4 tests: update `f` helper script to work on Python 3
Augie Fackler <augie@google.com>
parents: 32898
diff changeset
    76
            content = getattr(sys.stdin, 'buffer', sys.stdin).read()
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    77
            if opts.size:
36261
c69e78ef2b54 py3: port f to Python 3
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35472
diff changeset
    78
                facts.append(b'size=%d' % len(content))
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    79
        elif isdir:
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    80
            if opts.recurse or opts.type:
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    81
                dirfiles = glob.glob(f + '/*')
36261
c69e78ef2b54 py3: port f to Python 3
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35472
diff changeset
    82
                facts.append(b'directory with %d files' % len(dirfiles))
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    83
        elif opts.type:
36261
c69e78ef2b54 py3: port f to Python 3
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35472
diff changeset
    84
            facts.append(b'type unknown')
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    85
        if not isstdin:
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    86
            stat = os.lstat(f)
23911
593a5cd709a2 tests: teach f not to report directory size
Matt Mackall <mpm@selenic.com>
parents: 23860
diff changeset
    87
            if opts.size and not isdir:
36261
c69e78ef2b54 py3: port f to Python 3
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35472
diff changeset
    88
                facts.append(b'size=%d' % stat.st_size)
23912
7d0aa6269ece tests: teach f not to report symlink mode bits
Matt Mackall <mpm@selenic.com>
parents: 23911
diff changeset
    89
            if opts.mode and not islink:
36261
c69e78ef2b54 py3: port f to Python 3
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35472
diff changeset
    90
                facts.append(b'mode=%o' % (stat.st_mode & 0o777))
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    91
            if opts.links:
38364
06c85cbd6824 py3: use '%d' for os.stat_result.st_nlink instead of '%s'
Pulkit Goyal <7895pulkit@gmail.com>
parents: 36948
diff changeset
    92
                facts.append(b'links=%d' % stat.st_nlink)
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    93
            if opts.newer:
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    94
                # mtime might be in whole seconds so newer file might be same
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    95
                if stat.st_mtime >= os.stat(opts.newer).st_mtime:
43691
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
    96
                    facts.append(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
    97
                        b'newer than %s' % opts.newer.encode('utf8', 'replace')
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
    98
                    )
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
    99
                else:
43691
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   100
                    facts.append(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   101
                        b'older than %s' % opts.newer.encode('utf8', 'replace')
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   102
                    )
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   103
        if opts.md5 and content is not None:
29233
318534bb5dfd tests: make 'f' utility import hashlib unconditionally
Yuya Nishihara <yuya@tcha.org>
parents: 29230
diff changeset
   104
            h = hashlib.md5(content)
43691
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   105
            facts.append(b'md5=%s' % binascii.hexlify(h.digest())[: opts.bytes])
49556
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
   106
        if opts.raw_sha1 and content is not None:
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
   107
            h = hashlib.sha1(content)
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
   108
            facts.append(b'raw-sha1=%s' % h.digest()[: opts.bytes])
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   109
        if opts.sha1 and content is not None:
29233
318534bb5dfd tests: make 'f' utility import hashlib unconditionally
Yuya Nishihara <yuya@tcha.org>
parents: 29230
diff changeset
   110
            h = hashlib.sha1(content)
43691
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   111
            facts.append(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   112
                b'sha1=%s' % binascii.hexlify(h.digest())[: opts.bytes]
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   113
            )
35472
c1f7037c2ded tests: teach `f` to handle sha256 checksums
Matt Harbison <matt_harbison@yahoo.com>
parents: 35471
diff changeset
   114
        if opts.sha256 and content is not None:
c1f7037c2ded tests: teach `f` to handle sha256 checksums
Matt Harbison <matt_harbison@yahoo.com>
parents: 35471
diff changeset
   115
            h = hashlib.sha256(content)
43691
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   116
            facts.append(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   117
                b'sha256=%s' % binascii.hexlify(h.digest())[: opts.bytes]
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   118
            )
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   119
        if isstdin:
34270
3db2365d43e4 tests: update `f` helper script to work on Python 3
Augie Fackler <augie@google.com>
parents: 32898
diff changeset
   120
            outfile.write(b', '.join(facts) + b'\n')
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   121
        elif facts:
34270
3db2365d43e4 tests: update `f` helper script to work on Python 3
Augie Fackler <augie@google.com>
parents: 32898
diff changeset
   122
            outfile.write(b'%s: %s\n' % (f.encode('utf-8'), b', '.join(facts)))
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   123
        elif not quiet:
34270
3db2365d43e4 tests: update `f` helper script to work on Python 3
Augie Fackler <augie@google.com>
parents: 32898
diff changeset
   124
            outfile.write(b'%s:\n' % f.encode('utf-8'))
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   125
        if content is not None:
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   126
            chunk = content
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   127
            if not islink:
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   128
                if opts.lines:
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   129
                    if opts.lines >= 0:
43691
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   130
                        chunk = b''.join(chunk.splitlines(True)[: opts.lines])
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   131
                    else:
43691
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   132
                        chunk = b''.join(chunk.splitlines(True)[opts.lines :])
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   133
                if opts.bytes:
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   134
                    if opts.bytes >= 0:
43691
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   135
                        chunk = chunk[: opts.bytes]
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   136
                    else:
43691
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   137
                        chunk = chunk[opts.bytes :]
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   138
            if opts.hexdump:
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   139
                for i in range(0, len(chunk), 16):
43691
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   140
                    s = chunk[i : i + 16]
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   141
                    outfile.write(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   142
                        b'%04x: %-47s |%s|\n'
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   143
                        % (
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   144
                            i,
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   145
                            b' '.join(b'%02x' % ord(c) for c in iterbytes(s)),
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   146
                            re.sub(b'[^ -~]', b'.', s),
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   147
                        )
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   148
                    )
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   149
            if opts.dump:
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   150
                if not quiet:
34270
3db2365d43e4 tests: update `f` helper script to work on Python 3
Augie Fackler <augie@google.com>
parents: 32898
diff changeset
   151
                    outfile.write(b'>>>\n')
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   152
                outfile.write(chunk)
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   153
                if not quiet:
34270
3db2365d43e4 tests: update `f` helper script to work on Python 3
Augie Fackler <augie@google.com>
parents: 32898
diff changeset
   154
                    if chunk.endswith(b'\n'):
3db2365d43e4 tests: update `f` helper script to work on Python 3
Augie Fackler <augie@google.com>
parents: 32898
diff changeset
   155
                        outfile.write(b'<<<\n')
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   156
                    else:
34270
3db2365d43e4 tests: update `f` helper script to work on Python 3
Augie Fackler <augie@google.com>
parents: 32898
diff changeset
   157
                        outfile.write(b'\n<<< no trailing newline\n')
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   158
        if opts.recurse and dirfiles:
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   159
            assert not isstdin
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   160
            visit(opts, dirfiles, outfile)
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   161
43691
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   162
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   163
if __name__ == "__main__":
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   164
    parser = optparse.OptionParser("%prog [options] [filenames]")
43691
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   165
    parser.add_option(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   166
        "-t",
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   167
        "--type",
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   168
        action="store_true",
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   169
        help="show file type (file or directory)",
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   170
    )
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   171
    parser.add_option(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   172
        "-m", "--mode", action="store_true", help="show file mode"
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   173
    )
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   174
    parser.add_option(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   175
        "-l", "--links", action="store_true", help="show number of links"
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   176
    )
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   177
    parser.add_option(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   178
        "-s", "--size", action="store_true", help="show size of file"
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   179
    )
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   180
    parser.add_option(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   181
        "-n", "--newer", action="store", help="check if file is newer (or same)"
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   182
    )
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   183
    parser.add_option(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   184
        "-r", "--recurse", action="store_true", help="recurse into directories"
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   185
    )
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   186
    parser.add_option(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   187
        "-S",
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   188
        "--sha1",
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   189
        action="store_true",
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   190
        help="show sha1 hash of the content",
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   191
    )
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   192
    parser.add_option(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   193
        "",
49556
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
   194
        "--raw-sha1",
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
   195
        action="store_true",
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
   196
        help="show raw bytes of the sha1 hash of the content",
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
   197
    )
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
   198
    parser.add_option(
9172bd49cedc testlib: add `--raw-sha1` option to `f`
Raphaël Gomès <rgomes@octobus.net>
parents: 48875
diff changeset
   199
        "",
43691
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   200
        "--sha256",
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   201
        action="store_true",
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   202
        help="show sha256 hash of the content",
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   203
    )
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   204
    parser.add_option(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   205
        "-M", "--md5", action="store_true", help="show md5 hash of the content"
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   206
    )
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   207
    parser.add_option(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   208
        "-D", "--dump", action="store_true", help="dump file content"
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   209
    )
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   210
    parser.add_option(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   211
        "-H", "--hexdump", action="store_true", help="hexdump file content"
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   212
    )
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   213
    parser.add_option(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   214
        "-B", "--bytes", type="int", help="number of characters to dump"
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   215
    )
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   216
    parser.add_option(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   217
        "-L", "--lines", type="int", help="number of lines to dump"
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   218
    )
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   219
    parser.add_option(
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   220
        "-q", "--quiet", action="store_true", help="no default output"
47ef023d0165 black: blacken scripts
Gregory Szorc <gregory.szorc@gmail.com>
parents: 40306
diff changeset
   221
    )
23860
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   222
    (opts, filenames) = parser.parse_args(sys.argv[1:])
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   223
    if not filenames:
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   224
        filenames = ['-']
bead0c7b4f68 tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff changeset
   225
34270
3db2365d43e4 tests: update `f` helper script to work on Python 3
Augie Fackler <augie@google.com>
parents: 32898
diff changeset
   226
    visit(opts, filenames, getattr(sys.stdout, 'buffer', sys.stdout))