mercurial/util.py
author Bryan O'Sullivan <bos@serpentine.com>
Sat, 17 Sep 2005 00:27:27 -0700
changeset 1270 fc3b41570082
parent 1258 1945754e466b
child 1285 1546c2aa6b30
permissions -rw-r--r--
Switch to new syntax for .hgignore files. Here is the new syntax, in summary. Trailing white space is dropped. The escape character is "\". Comments start with #. Empty lines are skipped. Lines can be of the following formats: syntax: regexp # defaults following lines to non-rooted regexps syntax: glob # defaults following lines to non-rooted globs re:pattern # non-rooted regular expression glob:pattern # non-rooted glob pattern # pattern of the current default type The default pattern type is regexp, which is completely backwards compatible with the old hgignore syntax. In the dirstate class, the ignore method has been reworked to be based on the util.matcher function, by way of a new dirstate.hgignore method.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
     1
"""
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
     2
util.py - Mercurial utility functions and platform specfic implementations
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
     3
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
     4
 Copyright 2005 K. Thananchayan <thananck@yahoo.com>
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
     5
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
     6
This software may be used and distributed according to the terms
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
     7
of the GNU General Public License, incorporated herein by reference.
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
     8
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
     9
This contains helper routines that are independent of the SCM core and hide
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
    10
platform-specific details from the core.
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
    11
"""
419
28511fc21073 [PATCH] file seperator handling for the other 'OS'
mpm@selenic.com
parents:
diff changeset
    12
704
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
    13
import os, errno
724
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    14
from demandload import *
1258
1945754e466b Add file encoding/decoding support
mpm@selenic.com
parents: 1241
diff changeset
    15
demandload(globals(), "re cStringIO shutil popen2 threading")
1945754e466b Add file encoding/decoding support
mpm@selenic.com
parents: 1241
diff changeset
    16
1945754e466b Add file encoding/decoding support
mpm@selenic.com
parents: 1241
diff changeset
    17
def filter(s, cmd):
1945754e466b Add file encoding/decoding support
mpm@selenic.com
parents: 1241
diff changeset
    18
    "filter a string through a command that transforms its input to its output"
1945754e466b Add file encoding/decoding support
mpm@selenic.com
parents: 1241
diff changeset
    19
    (pout, pin) = popen2.popen2(cmd, -1, 'b')
1945754e466b Add file encoding/decoding support
mpm@selenic.com
parents: 1241
diff changeset
    20
    def writer():
1945754e466b Add file encoding/decoding support
mpm@selenic.com
parents: 1241
diff changeset
    21
        pin.write(s)
1945754e466b Add file encoding/decoding support
mpm@selenic.com
parents: 1241
diff changeset
    22
        pin.close()
1945754e466b Add file encoding/decoding support
mpm@selenic.com
parents: 1241
diff changeset
    23
1945754e466b Add file encoding/decoding support
mpm@selenic.com
parents: 1241
diff changeset
    24
    # we should use select instead on UNIX, but this will work on most
1945754e466b Add file encoding/decoding support
mpm@selenic.com
parents: 1241
diff changeset
    25
    # systems, including Windows
1945754e466b Add file encoding/decoding support
mpm@selenic.com
parents: 1241
diff changeset
    26
    w = threading.Thread(target=writer)
1945754e466b Add file encoding/decoding support
mpm@selenic.com
parents: 1241
diff changeset
    27
    w.start()
1945754e466b Add file encoding/decoding support
mpm@selenic.com
parents: 1241
diff changeset
    28
    f = pout.read()
1945754e466b Add file encoding/decoding support
mpm@selenic.com
parents: 1241
diff changeset
    29
    pout.close()
1945754e466b Add file encoding/decoding support
mpm@selenic.com
parents: 1241
diff changeset
    30
    w.join()
1945754e466b Add file encoding/decoding support
mpm@selenic.com
parents: 1241
diff changeset
    31
    return f
419
28511fc21073 [PATCH] file seperator handling for the other 'OS'
mpm@selenic.com
parents:
diff changeset
    32
1015
22571b8d35d3 Add automatic binary file detection to diff and export
mpm@selenic.com
parents: 917
diff changeset
    33
def binary(s):
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
    34
    """return true if a string is binary data using diff's heuristic"""
1015
22571b8d35d3 Add automatic binary file detection to diff and export
mpm@selenic.com
parents: 917
diff changeset
    35
    if s and '\0' in s[:4096]:
22571b8d35d3 Add automatic binary file detection to diff and export
mpm@selenic.com
parents: 917
diff changeset
    36
        return True
22571b8d35d3 Add automatic binary file detection to diff and export
mpm@selenic.com
parents: 917
diff changeset
    37
    return False
22571b8d35d3 Add automatic binary file detection to diff and export
mpm@selenic.com
parents: 917
diff changeset
    38
556
f6c6fa15ff70 Move dirstate.uniq to util.unique
mpm@selenic.com
parents: 521
diff changeset
    39
def unique(g):
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
    40
    """return the uniq elements of iterable g"""
556
f6c6fa15ff70 Move dirstate.uniq to util.unique
mpm@selenic.com
parents: 521
diff changeset
    41
    seen = {}
f6c6fa15ff70 Move dirstate.uniq to util.unique
mpm@selenic.com
parents: 521
diff changeset
    42
    for f in g:
f6c6fa15ff70 Move dirstate.uniq to util.unique
mpm@selenic.com
parents: 521
diff changeset
    43
        if f not in seen:
f6c6fa15ff70 Move dirstate.uniq to util.unique
mpm@selenic.com
parents: 521
diff changeset
    44
            seen[f] = 1
f6c6fa15ff70 Move dirstate.uniq to util.unique
mpm@selenic.com
parents: 521
diff changeset
    45
            yield f
f6c6fa15ff70 Move dirstate.uniq to util.unique
mpm@selenic.com
parents: 521
diff changeset
    46
870
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
    47
class Abort(Exception):
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
    48
    """Raised if a command needs to print an error and exit."""
508
42a660abaf75 [PATCH] Harden os.system
mpm@selenic.com
parents: 464
diff changeset
    49
724
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    50
def always(fn): return True
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    51
def never(fn): return False
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    52
1062
6d5a62a549fa pep-0008 cleanup
benoit.boissinot@ens-lyon.fr
parents: 1031
diff changeset
    53
def globre(pat, head='^', tail='$'):
724
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    54
    "convert a glob pattern into a regexp"
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    55
    i, n = 0, len(pat)
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    56
    res = ''
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    57
    group = False
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    58
    def peek(): return i < n and pat[i]
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    59
    while i < n:
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    60
        c = pat[i]
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    61
        i = i+1
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    62
        if c == '*':
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    63
            if peek() == '*':
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    64
                i += 1
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    65
                res += '.*'
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    66
            else:
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    67
                res += '[^/]*'
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    68
        elif c == '?':
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    69
            res += '.'
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    70
        elif c == '[':
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    71
            j = i
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    72
            if j < n and pat[j] in '!]':
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    73
                j += 1
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    74
            while j < n and pat[j] != ']':
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    75
                j += 1
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    76
            if j >= n:
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    77
                res += '\\['
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    78
            else:
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    79
                stuff = pat[i:j].replace('\\','\\\\')
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    80
                i = j + 1
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    81
                if stuff[0] == '!':
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    82
                    stuff = '^' + stuff[1:]
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    83
                elif stuff[0] == '^':
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    84
                    stuff = '\\' + stuff
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    85
                res = '%s[%s]' % (res, stuff)
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    86
        elif c == '{':
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    87
            group = True
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    88
            res += '(?:'
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    89
        elif c == '}' and group:
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    90
            res += ')'
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    91
            group = False
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    92
        elif c == ',' and group:
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    93
            res += '|'
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    94
        else:
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    95
            res += re.escape(c)
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    96
    return head + res + tail
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
    97
812
b65af904d6d7 Reduce the amount of stat traffic generated by a walk.
Bryan O'Sullivan <bos@serpentine.com>
parents: 782
diff changeset
    98
_globchars = {'[': 1, '{': 1, '*': 1, '?': 1}
b65af904d6d7 Reduce the amount of stat traffic generated by a walk.
Bryan O'Sullivan <bos@serpentine.com>
parents: 782
diff changeset
    99
884
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
   100
def pathto(n1, n2):
886
509de8ab6f31 Fix walk path handling on Windows
Bryan O'Sullivan <bos@serpentine.com>
parents: 884
diff changeset
   101
    '''return the relative path from one place to another.
509de8ab6f31 Fix walk path handling on Windows
Bryan O'Sullivan <bos@serpentine.com>
parents: 884
diff changeset
   102
    this returns a path in the form used by the local filesystem, not hg.'''
509de8ab6f31 Fix walk path handling on Windows
Bryan O'Sullivan <bos@serpentine.com>
parents: 884
diff changeset
   103
    if not n1: return localpath(n2)
509de8ab6f31 Fix walk path handling on Windows
Bryan O'Sullivan <bos@serpentine.com>
parents: 884
diff changeset
   104
    a, b = n1.split('/'), n2.split('/')
884
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
   105
    a.reverse(), b.reverse()
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
   106
    while a and b and a[-1] == b[-1]:
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
   107
        a.pop(), b.pop()
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
   108
    b.reverse()
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
   109
    return os.sep.join((['..'] * len(a)) + b)
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
   110
1081
8b7d63489db3 Change canonpath to not know about repo objects
mpm@selenic.com
parents: 1075
diff changeset
   111
def canonpath(root, cwd, myname):
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   112
    """return the canonical path of myname, given cwd and root"""
1081
8b7d63489db3 Change canonpath to not know about repo objects
mpm@selenic.com
parents: 1075
diff changeset
   113
    rootsep = root + os.sep
870
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   114
    name = myname
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   115
    if not name.startswith(os.sep):
1081
8b7d63489db3 Change canonpath to not know about repo objects
mpm@selenic.com
parents: 1075
diff changeset
   116
        name = os.path.join(root, cwd, name)
870
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   117
    name = os.path.normpath(name)
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   118
    if name.startswith(rootsep):
886
509de8ab6f31 Fix walk path handling on Windows
Bryan O'Sullivan <bos@serpentine.com>
parents: 884
diff changeset
   119
        return pconvert(name[len(rootsep):])
1081
8b7d63489db3 Change canonpath to not know about repo objects
mpm@selenic.com
parents: 1075
diff changeset
   120
    elif name == root:
870
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   121
        return ''
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   122
    else:
1081
8b7d63489db3 Change canonpath to not know about repo objects
mpm@selenic.com
parents: 1075
diff changeset
   123
        raise Abort('%s not under root' % myname)
897
fe30f5434b51 Fix bug with empty inc and exc
mpm@selenic.com
parents: 896
diff changeset
   124
1270
fc3b41570082 Switch to new syntax for .hgignore files.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1258
diff changeset
   125
def matcher(canonroot, cwd='', names=['.'], inc=[], exc=[], head=''):
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   126
    """build a function to match a set of file patterns
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   127
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   128
    arguments:
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   129
    canonroot - the canonical root of the tree you're matching against
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   130
    cwd - the current working directory, if relevant
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   131
    names - patterns to find
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   132
    inc - patterns to include
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   133
    exc - patterns to exclude
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   134
    head - a regex to prepend to patterns to control whether a match is rooted
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   135
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   136
    a pattern is one of:
1270
fc3b41570082 Switch to new syntax for .hgignore files.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1258
diff changeset
   137
    'glob:<rooted glob>'
fc3b41570082 Switch to new syntax for .hgignore files.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1258
diff changeset
   138
    're:<rooted regexp>'
fc3b41570082 Switch to new syntax for .hgignore files.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1258
diff changeset
   139
    'path:<rooted path>'
fc3b41570082 Switch to new syntax for .hgignore files.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1258
diff changeset
   140
    'relglob:<relative glob>'
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   141
    'relpath:<relative path>'
1270
fc3b41570082 Switch to new syntax for .hgignore files.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1258
diff changeset
   142
    'relre:<relative regexp>'
fc3b41570082 Switch to new syntax for .hgignore files.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1258
diff changeset
   143
    '<rooted path or regexp>'
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   144
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   145
    returns:
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   146
    a 3-tuple containing
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   147
    - list of explicit non-pattern names passed in
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   148
    - a bool match(filename) function
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   149
    - a bool indicating if any patterns were passed in
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   150
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   151
    todo:
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   152
    make head regex a rooted bool
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   153
    """
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   154
820
89985a1b3427 Clean up walk and changes code to use normalised names properly.
Bryan O'Sullivan <bos@serpentine.com>
parents: 814
diff changeset
   155
    def patkind(name):
1270
fc3b41570082 Switch to new syntax for .hgignore files.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1258
diff changeset
   156
        for prefix in 're', 'glob', 'path', 'relglob', 'relpath', 'relre':
fc3b41570082 Switch to new syntax for .hgignore files.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1258
diff changeset
   157
            if name.startswith(prefix + ':'): return name.split(':', 1)
812
b65af904d6d7 Reduce the amount of stat traffic generated by a walk.
Bryan O'Sullivan <bos@serpentine.com>
parents: 782
diff changeset
   158
        for c in name:
820
89985a1b3427 Clean up walk and changes code to use normalised names properly.
Bryan O'Sullivan <bos@serpentine.com>
parents: 814
diff changeset
   159
            if c in _globchars: return 'glob', name
89985a1b3427 Clean up walk and changes code to use normalised names properly.
Bryan O'Sullivan <bos@serpentine.com>
parents: 814
diff changeset
   160
        return 'relpath', name
89985a1b3427 Clean up walk and changes code to use normalised names properly.
Bryan O'Sullivan <bos@serpentine.com>
parents: 814
diff changeset
   161
888
e7a943e8c52b Fix up handling of regexp paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 886
diff changeset
   162
    def regex(kind, name, tail):
742
092937de2ad7 Refactor matchpats and walk
mpm@selenic.com
parents: 740
diff changeset
   163
        '''convert a pattern into a regular expression'''
820
89985a1b3427 Clean up walk and changes code to use normalised names properly.
Bryan O'Sullivan <bos@serpentine.com>
parents: 814
diff changeset
   164
        if kind == 're':
89985a1b3427 Clean up walk and changes code to use normalised names properly.
Bryan O'Sullivan <bos@serpentine.com>
parents: 814
diff changeset
   165
            return name
89985a1b3427 Clean up walk and changes code to use normalised names properly.
Bryan O'Sullivan <bos@serpentine.com>
parents: 814
diff changeset
   166
        elif kind == 'path':
888
e7a943e8c52b Fix up handling of regexp paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 886
diff changeset
   167
            return '^' + re.escape(name) + '(?:/|$)'
1270
fc3b41570082 Switch to new syntax for .hgignore files.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1258
diff changeset
   168
        elif kind == 'relglob':
fc3b41570082 Switch to new syntax for .hgignore files.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1258
diff changeset
   169
            return head + globre(name, '(?:|.*/)', tail)
888
e7a943e8c52b Fix up handling of regexp paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 886
diff changeset
   170
        elif kind == 'relpath':
e7a943e8c52b Fix up handling of regexp paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 886
diff changeset
   171
            return head + re.escape(name) + tail
1270
fc3b41570082 Switch to new syntax for .hgignore files.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1258
diff changeset
   172
        elif kind == 'relre':
fc3b41570082 Switch to new syntax for .hgignore files.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1258
diff changeset
   173
            if name.startswith('^'):
fc3b41570082 Switch to new syntax for .hgignore files.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1258
diff changeset
   174
                return name
fc3b41570082 Switch to new syntax for .hgignore files.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1258
diff changeset
   175
            return '.*' + name
742
092937de2ad7 Refactor matchpats and walk
mpm@selenic.com
parents: 740
diff changeset
   176
        return head + globre(name, '', tail)
092937de2ad7 Refactor matchpats and walk
mpm@selenic.com
parents: 740
diff changeset
   177
092937de2ad7 Refactor matchpats and walk
mpm@selenic.com
parents: 740
diff changeset
   178
    def matchfn(pats, tail):
092937de2ad7 Refactor matchpats and walk
mpm@selenic.com
parents: 740
diff changeset
   179
        """build a matching function from a set of patterns"""
092937de2ad7 Refactor matchpats and walk
mpm@selenic.com
parents: 740
diff changeset
   180
        if pats:
888
e7a943e8c52b Fix up handling of regexp paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 886
diff changeset
   181
            pat = '(?:%s)' % '|'.join([regex(k, p, tail) for (k, p) in pats])
742
092937de2ad7 Refactor matchpats and walk
mpm@selenic.com
parents: 740
diff changeset
   182
            return re.compile(pat).match
092937de2ad7 Refactor matchpats and walk
mpm@selenic.com
parents: 740
diff changeset
   183
820
89985a1b3427 Clean up walk and changes code to use normalised names properly.
Bryan O'Sullivan <bos@serpentine.com>
parents: 814
diff changeset
   184
    def globprefix(pat):
89985a1b3427 Clean up walk and changes code to use normalised names properly.
Bryan O'Sullivan <bos@serpentine.com>
parents: 814
diff changeset
   185
        '''return the non-glob prefix of a path, e.g. foo/* -> foo'''
89985a1b3427 Clean up walk and changes code to use normalised names properly.
Bryan O'Sullivan <bos@serpentine.com>
parents: 814
diff changeset
   186
        root = []
89985a1b3427 Clean up walk and changes code to use normalised names properly.
Bryan O'Sullivan <bos@serpentine.com>
parents: 814
diff changeset
   187
        for p in pat.split(os.sep):
89985a1b3427 Clean up walk and changes code to use normalised names properly.
Bryan O'Sullivan <bos@serpentine.com>
parents: 814
diff changeset
   188
            if patkind(p)[0] == 'glob': break
89985a1b3427 Clean up walk and changes code to use normalised names properly.
Bryan O'Sullivan <bos@serpentine.com>
parents: 814
diff changeset
   189
            root.append(p)
886
509de8ab6f31 Fix walk path handling on Windows
Bryan O'Sullivan <bos@serpentine.com>
parents: 884
diff changeset
   190
        return '/'.join(root)
820
89985a1b3427 Clean up walk and changes code to use normalised names properly.
Bryan O'Sullivan <bos@serpentine.com>
parents: 814
diff changeset
   191
870
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   192
    pats = []
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   193
    files = []
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   194
    roots = []
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   195
    for kind, name in map(patkind, names):
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   196
        if kind in ('glob', 'relpath'):
1081
8b7d63489db3 Change canonpath to not know about repo objects
mpm@selenic.com
parents: 1075
diff changeset
   197
            name = canonpath(canonroot, cwd, name)
870
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   198
            if name == '':
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   199
                kind, name = 'glob', '**'
888
e7a943e8c52b Fix up handling of regexp paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 886
diff changeset
   200
        if kind in ('glob', 'path', 're'):
e7a943e8c52b Fix up handling of regexp paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 886
diff changeset
   201
            pats.append((kind, name))
870
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   202
        if kind == 'glob':
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   203
            root = globprefix(name)
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   204
            if root: roots.append(root)
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   205
        elif kind == 'relpath':
888
e7a943e8c52b Fix up handling of regexp paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 886
diff changeset
   206
            files.append((kind, name))
870
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   207
            roots.append(name)
897
fe30f5434b51 Fix bug with empty inc and exc
mpm@selenic.com
parents: 896
diff changeset
   208
820
89985a1b3427 Clean up walk and changes code to use normalised names properly.
Bryan O'Sullivan <bos@serpentine.com>
parents: 814
diff changeset
   209
    patmatch = matchfn(pats, '$') or always
89985a1b3427 Clean up walk and changes code to use normalised names properly.
Bryan O'Sullivan <bos@serpentine.com>
parents: 814
diff changeset
   210
    filematch = matchfn(files, '(?:/|$)') or always
897
fe30f5434b51 Fix bug with empty inc and exc
mpm@selenic.com
parents: 896
diff changeset
   211
    incmatch = always
fe30f5434b51 Fix bug with empty inc and exc
mpm@selenic.com
parents: 896
diff changeset
   212
    if inc:
fe30f5434b51 Fix bug with empty inc and exc
mpm@selenic.com
parents: 896
diff changeset
   213
        incmatch = matchfn(map(patkind, inc), '(?:/|$)')
fe30f5434b51 Fix bug with empty inc and exc
mpm@selenic.com
parents: 896
diff changeset
   214
    excmatch = lambda fn: False
fe30f5434b51 Fix bug with empty inc and exc
mpm@selenic.com
parents: 896
diff changeset
   215
    if exc:
fe30f5434b51 Fix bug with empty inc and exc
mpm@selenic.com
parents: 896
diff changeset
   216
        excmatch = matchfn(map(patkind, exc), '(?:/|$)')
742
092937de2ad7 Refactor matchpats and walk
mpm@selenic.com
parents: 740
diff changeset
   217
1031
503aaf19a040 Rewrite log command. New version is faster and more featureful.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1015
diff changeset
   218
    return (roots,
503aaf19a040 Rewrite log command. New version is faster and more featureful.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1015
diff changeset
   219
            lambda fn: (incmatch(fn) and not excmatch(fn) and
503aaf19a040 Rewrite log command. New version is faster and more featureful.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1015
diff changeset
   220
                        (fn.endswith('/') or
503aaf19a040 Rewrite log command. New version is faster and more featureful.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1015
diff changeset
   221
                         (not pats and not files) or
503aaf19a040 Rewrite log command. New version is faster and more featureful.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1015
diff changeset
   222
                         (pats and patmatch(fn)) or
503aaf19a040 Rewrite log command. New version is faster and more featureful.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1015
diff changeset
   223
                         (files and filematch(fn)))),
503aaf19a040 Rewrite log command. New version is faster and more featureful.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1015
diff changeset
   224
            (inc or exc or (pats and pats != [('glob', '**')])) and True)
742
092937de2ad7 Refactor matchpats and walk
mpm@selenic.com
parents: 740
diff changeset
   225
521
0fb8ade0f756 [PATCH] Fix use of util.CommandError
mpm@selenic.com
parents: 515
diff changeset
   226
def system(cmd, errprefix=None):
508
42a660abaf75 [PATCH] Harden os.system
mpm@selenic.com
parents: 464
diff changeset
   227
    """execute a shell command that must succeed"""
42a660abaf75 [PATCH] Harden os.system
mpm@selenic.com
parents: 464
diff changeset
   228
    rc = os.system(cmd)
42a660abaf75 [PATCH] Harden os.system
mpm@selenic.com
parents: 464
diff changeset
   229
    if rc:
521
0fb8ade0f756 [PATCH] Fix use of util.CommandError
mpm@selenic.com
parents: 515
diff changeset
   230
        errmsg = "%s %s" % (os.path.basename(cmd.split(None, 1)[0]),
0fb8ade0f756 [PATCH] Fix use of util.CommandError
mpm@selenic.com
parents: 515
diff changeset
   231
                            explain_exit(rc)[0])
0fb8ade0f756 [PATCH] Fix use of util.CommandError
mpm@selenic.com
parents: 515
diff changeset
   232
        if errprefix:
0fb8ade0f756 [PATCH] Fix use of util.CommandError
mpm@selenic.com
parents: 515
diff changeset
   233
            errmsg = "%s: %s" % (errprefix, errmsg)
870
a82eae840447 Teach walk code about absolute paths.
Bryan O'Sullivan <bos@serpentine.com>
parents: 869
diff changeset
   234
        raise Abort(errmsg)
508
42a660abaf75 [PATCH] Harden os.system
mpm@selenic.com
parents: 464
diff changeset
   235
421
43b8da7420a9 [PATCH] rename under the other OS
mpm@selenic.com
parents: 419
diff changeset
   236
def rename(src, dst):
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   237
    """forcibly rename a file"""
421
43b8da7420a9 [PATCH] rename under the other OS
mpm@selenic.com
parents: 419
diff changeset
   238
    try:
43b8da7420a9 [PATCH] rename under the other OS
mpm@selenic.com
parents: 419
diff changeset
   239
        os.rename(src, dst)
43b8da7420a9 [PATCH] rename under the other OS
mpm@selenic.com
parents: 419
diff changeset
   240
    except:
43b8da7420a9 [PATCH] rename under the other OS
mpm@selenic.com
parents: 419
diff changeset
   241
        os.unlink(dst)
43b8da7420a9 [PATCH] rename under the other OS
mpm@selenic.com
parents: 419
diff changeset
   242
        os.rename(src, dst)
43b8da7420a9 [PATCH] rename under the other OS
mpm@selenic.com
parents: 419
diff changeset
   243
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   244
def copyfiles(src, dst, hardlink=None):
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   245
    """Copy a directory tree using hardlinks if possible"""
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   246
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   247
    if hardlink is None:
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   248
        hardlink = (os.stat(src).st_dev ==
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   249
                    os.stat(os.path.dirname(dst)).st_dev)
698
df78d8ccac4c Use python function instead of external 'cp' command when cloning repos.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 667
diff changeset
   250
1207
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
   251
    if os.path.isdir(src):
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
   252
        os.mkdir(dst)
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
   253
        for name in os.listdir(src):
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
   254
            srcname = os.path.join(src, name)
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
   255
            dstname = os.path.join(dst, name)
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   256
            copyfiles(srcname, dstname, hardlink)
1207
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
   257
    else:
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   258
        if hardlink:
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   259
            try:
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   260
                os_link(src, dst)
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   261
            except:
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   262
                hardlink = False
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   263
                shutil.copy2(src, dst)
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   264
        else:
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   265
            shutil.copy2(src, dst)
698
df78d8ccac4c Use python function instead of external 'cp' command when cloning repos.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 667
diff changeset
   266
1090
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   267
def opener(base):
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   268
    """
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   269
    return a function that opens files relative to base
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   270
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   271
    this function is used to hide the details of COW semantics and
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   272
    remote file access from higher level code.
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   273
    """
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   274
    p = base
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   275
    def o(path, mode="r"):
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   276
        f = os.path.join(p, path)
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   277
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   278
        mode += "b" # for that other OS
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   279
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   280
        if mode[0] != "r":
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   281
            try:
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   282
                nlink = nlinks(f)
1090
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   283
            except OSError:
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   284
                d = os.path.dirname(f)
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   285
                if not os.path.isdir(d):
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   286
                    os.makedirs(d)
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   287
            else:
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   288
                if nlink > 1:
1090
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   289
                    file(f + ".tmp", "wb").write(file(f, "rb").read())
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   290
                    rename(f+".tmp", f)
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   291
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   292
        return file(f, mode)
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   293
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   294
    return o
1bca39b85615 Move opener to utils
mpm@selenic.com
parents: 1082
diff changeset
   295
704
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   296
def _makelock_file(info, pathname):
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   297
    ld = os.open(pathname, os.O_CREAT | os.O_WRONLY | os.O_EXCL)
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   298
    os.write(ld, info)
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   299
    os.close(ld)
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   300
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   301
def _readlock_file(pathname):
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   302
    return file(pathname).read()
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   303
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   304
def nlinks(pathname):
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   305
    """Return number of hardlinks for the given file."""
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   306
    return os.stat(pathname).st_nlink
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   307
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   308
if hasattr(os, 'link'):
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   309
    os_link = os.link
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   310
else:
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   311
    def os_link(src, dst):
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   312
        raise OSError(0, "Hardlinks not supported")
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   313
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   314
# Platform specific variants
419
28511fc21073 [PATCH] file seperator handling for the other 'OS'
mpm@selenic.com
parents:
diff changeset
   315
if os.name == 'nt':
461
9ae0034f2772 [PATCH] /dev/null for other OS
mpm@selenic.com
parents: 441
diff changeset
   316
    nulldev = 'NUL:'
9ae0034f2772 [PATCH] /dev/null for other OS
mpm@selenic.com
parents: 441
diff changeset
   317
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   318
    try: # ActivePython can create hard links using win32file module
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   319
        import win32file
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   320
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   321
        def os_link(src, dst): # NB will only succeed on NTFS
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   322
            win32file.CreateHardLink(dst, src)
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   323
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   324
        def nlinks(pathname):
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   325
            """Return number of hardlinks for the given file."""
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   326
            try:
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   327
                fh = win32file.CreateFile(pathname,
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   328
                    win32file.GENERIC_READ, win32file.FILE_SHARE_READ,
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   329
                    None, win32file.OPEN_EXISTING, 0, None)
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   330
                res = win32file.GetFileInformationByHandle(fh)
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   331
                fh.Close()
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   332
                return res[7]
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   333
            except:
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   334
                return os.stat(pathname).st_nlink
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   335
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   336
    except ImportError:
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   337
        pass
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
   338
441
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   339
    def is_exec(f, last):
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   340
        return last
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   341
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   342
    def set_exec(f, mode):
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   343
        pass
515
03f27b1381f9 Whitespace cleanups
mpm@selenic.com
parents: 508
diff changeset
   344
419
28511fc21073 [PATCH] file seperator handling for the other 'OS'
mpm@selenic.com
parents:
diff changeset
   345
    def pconvert(path):
28511fc21073 [PATCH] file seperator handling for the other 'OS'
mpm@selenic.com
parents:
diff changeset
   346
        return path.replace("\\", "/")
422
10c43444a38e [PATCH] Enables lock work under the other 'OS'
mpm@selenic.com
parents: 421
diff changeset
   347
886
509de8ab6f31 Fix walk path handling on Windows
Bryan O'Sullivan <bos@serpentine.com>
parents: 884
diff changeset
   348
    def localpath(path):
509de8ab6f31 Fix walk path handling on Windows
Bryan O'Sullivan <bos@serpentine.com>
parents: 884
diff changeset
   349
        return path.replace('/', '\\')
509de8ab6f31 Fix walk path handling on Windows
Bryan O'Sullivan <bos@serpentine.com>
parents: 884
diff changeset
   350
509de8ab6f31 Fix walk path handling on Windows
Bryan O'Sullivan <bos@serpentine.com>
parents: 884
diff changeset
   351
    def normpath(path):
509de8ab6f31 Fix walk path handling on Windows
Bryan O'Sullivan <bos@serpentine.com>
parents: 884
diff changeset
   352
        return pconvert(os.path.normpath(path))
509de8ab6f31 Fix walk path handling on Windows
Bryan O'Sullivan <bos@serpentine.com>
parents: 884
diff changeset
   353
704
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   354
    makelock = _makelock_file
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   355
    readlock = _readlock_file
461
9ae0034f2772 [PATCH] /dev/null for other OS
mpm@selenic.com
parents: 441
diff changeset
   356
782
cdb9e95b2fab Provided platform dependent implementations for explain_exit
thananck@yahoo.com
parents: 742
diff changeset
   357
    def explain_exit(code):
cdb9e95b2fab Provided platform dependent implementations for explain_exit
thananck@yahoo.com
parents: 742
diff changeset
   358
        return "exited with status %d" % code, code
cdb9e95b2fab Provided platform dependent implementations for explain_exit
thananck@yahoo.com
parents: 742
diff changeset
   359
419
28511fc21073 [PATCH] file seperator handling for the other 'OS'
mpm@selenic.com
parents:
diff changeset
   360
else:
461
9ae0034f2772 [PATCH] /dev/null for other OS
mpm@selenic.com
parents: 441
diff changeset
   361
    nulldev = '/dev/null'
9ae0034f2772 [PATCH] /dev/null for other OS
mpm@selenic.com
parents: 441
diff changeset
   362
441
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   363
    def is_exec(f, last):
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
   364
        """check whether a file is executable"""
441
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   365
        return (os.stat(f).st_mode & 0100 != 0)
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   366
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   367
    def set_exec(f, mode):
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   368
        s = os.stat(f).st_mode
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   369
        if (s & 0100 != 0) == mode:
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   370
            return
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   371
        if mode:
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   372
            # Turn on +x for every +r bit when making a file executable
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   373
            # and obey umask.
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   374
            umask = os.umask(0)
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   375
            os.umask(umask)
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   376
            os.chmod(f, s | (s & 0444) >> 2 & ~umask)
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   377
        else:
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   378
            os.chmod(f, s & 0666)
e8af362cfb01 Permission handling for the other OS
mpm@selenic.com
parents: 422
diff changeset
   379
419
28511fc21073 [PATCH] file seperator handling for the other 'OS'
mpm@selenic.com
parents:
diff changeset
   380
    def pconvert(path):
28511fc21073 [PATCH] file seperator handling for the other 'OS'
mpm@selenic.com
parents:
diff changeset
   381
        return path
28511fc21073 [PATCH] file seperator handling for the other 'OS'
mpm@selenic.com
parents:
diff changeset
   382
886
509de8ab6f31 Fix walk path handling on Windows
Bryan O'Sullivan <bos@serpentine.com>
parents: 884
diff changeset
   383
    def localpath(path):
509de8ab6f31 Fix walk path handling on Windows
Bryan O'Sullivan <bos@serpentine.com>
parents: 884
diff changeset
   384
        return path
509de8ab6f31 Fix walk path handling on Windows
Bryan O'Sullivan <bos@serpentine.com>
parents: 884
diff changeset
   385
509de8ab6f31 Fix walk path handling on Windows
Bryan O'Sullivan <bos@serpentine.com>
parents: 884
diff changeset
   386
    normpath = os.path.normpath
509de8ab6f31 Fix walk path handling on Windows
Bryan O'Sullivan <bos@serpentine.com>
parents: 884
diff changeset
   387
422
10c43444a38e [PATCH] Enables lock work under the other 'OS'
mpm@selenic.com
parents: 421
diff changeset
   388
    def makelock(info, pathname):
704
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   389
        try:
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   390
            os.symlink(info, pathname)
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   391
        except OSError, why:
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   392
            if why.errno == errno.EEXIST:
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   393
                raise
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   394
            else:
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   395
                _makelock_file(info, pathname)
422
10c43444a38e [PATCH] Enables lock work under the other 'OS'
mpm@selenic.com
parents: 421
diff changeset
   396
10c43444a38e [PATCH] Enables lock work under the other 'OS'
mpm@selenic.com
parents: 421
diff changeset
   397
    def readlock(pathname):
704
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   398
        try:
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   399
            return os.readlink(pathname)
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   400
        except OSError, why:
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   401
            if why.errno == errno.EINVAL:
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   402
                return _readlock_file(pathname)
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   403
            else:
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
   404
                raise
782
cdb9e95b2fab Provided platform dependent implementations for explain_exit
thananck@yahoo.com
parents: 742
diff changeset
   405
cdb9e95b2fab Provided platform dependent implementations for explain_exit
thananck@yahoo.com
parents: 742
diff changeset
   406
    def explain_exit(code):
cdb9e95b2fab Provided platform dependent implementations for explain_exit
thananck@yahoo.com
parents: 742
diff changeset
   407
        """return a 2-tuple (desc, code) describing a process's status"""
cdb9e95b2fab Provided platform dependent implementations for explain_exit
thananck@yahoo.com
parents: 742
diff changeset
   408
        if os.WIFEXITED(code):
cdb9e95b2fab Provided platform dependent implementations for explain_exit
thananck@yahoo.com
parents: 742
diff changeset
   409
            val = os.WEXITSTATUS(code)
cdb9e95b2fab Provided platform dependent implementations for explain_exit
thananck@yahoo.com
parents: 742
diff changeset
   410
            return "exited with status %d" % val, val
cdb9e95b2fab Provided platform dependent implementations for explain_exit
thananck@yahoo.com
parents: 742
diff changeset
   411
        elif os.WIFSIGNALED(code):
cdb9e95b2fab Provided platform dependent implementations for explain_exit
thananck@yahoo.com
parents: 742
diff changeset
   412
            val = os.WTERMSIG(code)
cdb9e95b2fab Provided platform dependent implementations for explain_exit
thananck@yahoo.com
parents: 742
diff changeset
   413
            return "killed by signal %d" % val, val
cdb9e95b2fab Provided platform dependent implementations for explain_exit
thananck@yahoo.com
parents: 742
diff changeset
   414
        elif os.WIFSTOPPED(code):
912
302f83b85054 Minor tweak: os.STOPSIG -> os.WSTOPSIG. Pychecker spotted this one.
mark.williamson@cl.cam.ac.uk
parents: 897
diff changeset
   415
            val = os.WSTOPSIG(code)
782
cdb9e95b2fab Provided platform dependent implementations for explain_exit
thananck@yahoo.com
parents: 742
diff changeset
   416
            return "stopped by signal %d" % val, val
cdb9e95b2fab Provided platform dependent implementations for explain_exit
thananck@yahoo.com
parents: 742
diff changeset
   417
        raise ValueError("invalid exit code")
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   418
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   419
class chunkbuffer(object):
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   420
    """Allow arbitrary sized chunks of data to be efficiently read from an
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   421
    iterator over chunks of arbitrary size."""
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
   422
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   423
    def __init__(self, in_iter, targetsize = 2**16):
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   424
        """in_iter is the iterator that's iterating over the input chunks.
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   425
        targetsize is how big a buffer to try to maintain."""
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   426
        self.in_iter = iter(in_iter)
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   427
        self.buf = ''
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   428
        self.targetsize = int(targetsize)
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
   429
        if self.targetsize <= 0:
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
   430
            raise ValueError("targetsize must be greater than 0, was %d" %
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
   431
                             targetsize)
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   432
        self.iterempty = False
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
   433
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   434
    def fillbuf(self):
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
   435
        """Ignore target size; read every chunk from iterator until empty."""
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   436
        if not self.iterempty:
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   437
            collector = cStringIO.StringIO()
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   438
            collector.write(self.buf)
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   439
            for ch in self.in_iter:
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   440
                collector.write(ch)
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   441
            self.buf = collector.getvalue()
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   442
            self.iterempty = True
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   443
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   444
    def read(self, l):
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
   445
        """Read L bytes of data from the iterator of chunks of data.
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
   446
	Returns less than L bytes if the iterator runs dry."""
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   447
        if l > len(self.buf) and not self.iterempty:
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   448
            # Clamp to a multiple of self.targetsize
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   449
            targetsize = self.targetsize * ((l // self.targetsize) + 1)
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   450
            collector = cStringIO.StringIO()
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   451
            collector.write(self.buf)
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   452
            collected = len(self.buf)
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   453
            for chunk in self.in_iter:
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   454
                collector.write(chunk)
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   455
                collected += len(chunk)
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   456
                if collected >= targetsize:
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   457
                    break
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   458
            if collected < targetsize:
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   459
                self.iterempty = True
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   460
            self.buf = collector.getvalue()
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
   461
        s, self.buf = self.buf[:l], buffer(self.buf, l)
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   462
        return s
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   463
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   464
def filechunkiter(f, size = 65536):
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
   465
    """Create a generator that produces all the data in the file size
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
   466
    (default 65536) bytes at a time.  Chunks may be less than size
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
   467
    bytes if the chunk is the last chunk in the file, or the file is a
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
   468
    socket or some other type of file that sometimes reads less data
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
   469
    than is requested."""
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   470
    s = f.read(size)
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   471
    while len(s) >= 0:
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   472
        yield s
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
   473
        s = f.read(size)