mercurial/util.py
author Pierre-Yves David <pierre-yves.david@octobus.net>
Tue, 26 Mar 2024 08:43:20 +0000
changeset 51546 f376c4d5c4b6
parent 51430 187c5769a629
child 51549 a452807df09b
child 51610 6c39edd1d348
permissions -rw-r--r--
perf-stream-locked-section: advertise the right version key in the help As the v3 format is still experimental, its key is "v3-exp". The help text was not pointing that out. (we also fix `perf::stream-generate` in the process)
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
17515
b5b38d21fe99 spelling: specific
timeless@mozdev.org
parents: 17237
diff changeset
     1
# util.py - Mercurial utility functions and platform specific implementations
8226
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
     2
#
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
     3
#  Copyright 2005 K. Thananchayan <thananck@yahoo.com>
46819
d4ba4d51f85f contributor: change mentions of mpm to olivia
Raphaël Gomès <rgomes@octobus.net>
parents: 46675
diff changeset
     4
#  Copyright 2005-2007 Olivia Mackall <olivia@selenic.com>
8226
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
     5
#  Copyright 2006 Vadim Gelfer <vadim.gelfer@gmail.com>
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
     6
#
8b2cd04a6e97 put license and copyright info into comment blocks
Martin Geisler <mg@lazybytes.net>
parents: 8225
diff changeset
     7
# This software may be used and distributed according to the terms of the
10263
25e572394f5c Update license to GPLv2+
Matt Mackall <mpm@selenic.com>
parents: 9996
diff changeset
     8
# GNU General Public License version 2 or any later version.
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
     9
17515
b5b38d21fe99 spelling: specific
timeless@mozdev.org
parents: 17237
diff changeset
    10
"""Mercurial utility functions and platform specific implementations.
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
    11
8227
0a9542703300 turn some comments back into module docstrings
Martin Geisler <mg@lazybytes.net>
parents: 8226
diff changeset
    12
This contains helper routines that are independent of the SCM core and
0a9542703300 turn some comments back into module docstrings
Martin Geisler <mg@lazybytes.net>
parents: 8226
diff changeset
    13
hide platform-specific details from the core.
1082
ce96e316278a Update util.py docstrings, fix walk test
mpm@selenic.com
parents: 1081
diff changeset
    14
"""
419
28511fc21073 [PATCH] file seperator handling for the other 'OS'
mpm@selenic.com
parents:
diff changeset
    15
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
    16
33793
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
    17
import abc
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
    18
import collections
33446
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
    19
import contextlib
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
    20
import errno
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
    21
import gc
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
    22
import hashlib
48873
5aafc3c5bdec py3: use io.BytesIO directly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48870
diff changeset
    23
import io
34554
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
    24
import itertools
45011
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
    25
import locale
34295
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34139
diff changeset
    26
import mmap
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
    27
import os
48993
225659936fff util: restore the util.pickle symbol
Matt Harbison <matt_harbison@yahoo.com>
parents: 48946
diff changeset
    28
import pickle  # provides util.pickle symbol
21907
7e5dfa00e3c2 util: rename 're' to 'remod'
Siddharth Agarwal <sid0@fb.com>
parents: 21857
diff changeset
    29
import re as remod
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
    30
import shutil
30418
1156ec81f709 util: improve iterfile so it chooses code path wisely
Jun Wu <quark@fb.com>
parents: 30417
diff changeset
    31
import stat
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
    32
import sys
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
    33
import time
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
    34
import traceback
31950
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
    35
import warnings
3769
96095d9ff1f8 Add encoding detection
Matt Mackall <mpm@selenic.com>
parents: 3767
diff changeset
    36
51285
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50929
diff changeset
    37
from typing import (
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50929
diff changeset
    38
    Iterable,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50929
diff changeset
    39
    Iterator,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50929
diff changeset
    40
    List,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50929
diff changeset
    41
    Optional,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50929
diff changeset
    42
    Tuple,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50929
diff changeset
    43
)
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50929
diff changeset
    44
47012
d55b71393907 node: replace nullid and friends with nodeconstants class
Joerg Sonnenberger <joerg@bec.de>
parents: 46926
diff changeset
    45
from .node import hex
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
    46
from .thirdparty import attr
43087
66f2cc210a29 py3: manually import pycompat.setattr where it is needed
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43085
diff changeset
    47
from .pycompat import (
66f2cc210a29 py3: manually import pycompat.setattr where it is needed
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43085
diff changeset
    48
    open,
66f2cc210a29 py3: manually import pycompat.setattr where it is needed
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43085
diff changeset
    49
)
39259
e00123f63410 util: make timedcm require the label (API)
Augie Fackler <augie@google.com>
parents: 39258
diff changeset
    50
from hgdemandimport import tracing
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
    51
from . import (
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
    52
    encoding,
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
    53
    error,
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
    54
    i18n,
32367
a9c71d578a1c osutil: switch to policy importer
Yuya Nishihara <yuya@tcha.org>
parents: 32306
diff changeset
    55
    policy,
28818
6041fb8f2da8 pycompat: add empty and queue to handle py3 divergence
timeless <timeless@mozdev.org>
parents: 28497
diff changeset
    56
    pycompat,
34467
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34435
diff changeset
    57
    urllibcompat,
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
    58
)
37083
f99d64e8a4e4 stringutil: move generic string helpers to new module
Yuya Nishihara <yuya@tcha.org>
parents: 37082
diff changeset
    59
from .utils import (
42041
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41834
diff changeset
    60
    compression,
44060
a61287a95dc3 core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents: 44035
diff changeset
    61
    hashutil,
37118
5be286db5fb5 procutil: move process/executable management functions to new module
Yuya Nishihara <yuya@tcha.org>
parents: 37117
diff changeset
    62
    procutil,
37083
f99d64e8a4e4 stringutil: move generic string helpers to new module
Yuya Nishihara <yuya@tcha.org>
parents: 37082
diff changeset
    63
    stringutil,
f99d64e8a4e4 stringutil: move generic string helpers to new module
Yuya Nishihara <yuya@tcha.org>
parents: 37082
diff changeset
    64
)
3769
96095d9ff1f8 Add encoding detection
Matt Mackall <mpm@selenic.com>
parents: 3767
diff changeset
    65
51285
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50929
diff changeset
    66
# keeps pyflakes happy
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50929
diff changeset
    67
assert [
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50929
diff changeset
    68
    Iterable,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50929
diff changeset
    69
    Iterator,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50929
diff changeset
    70
    List,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50929
diff changeset
    71
    Optional,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50929
diff changeset
    72
    Tuple,
9d3721552b6c pytype: import typing directly
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50929
diff changeset
    73
]
46645
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46113
diff changeset
    74
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46113
diff changeset
    75
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
    76
base85 = policy.importmod('base85')
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
    77
osutil = policy.importmod('osutil')
32367
a9c71d578a1c osutil: switch to policy importer
Yuya Nishihara <yuya@tcha.org>
parents: 32306
diff changeset
    78
32201
4462a981e8df base85: proxy through util module
Yuya Nishihara <yuya@tcha.org>
parents: 32154
diff changeset
    79
b85decode = base85.b85decode
4462a981e8df base85: proxy through util module
Yuya Nishihara <yuya@tcha.org>
parents: 32154
diff changeset
    80
b85encode = base85.b85encode
4462a981e8df base85: proxy through util module
Yuya Nishihara <yuya@tcha.org>
parents: 32154
diff changeset
    81
31934
12aca6770046 util: make cookielib module available
Gregory Szorc <gregory.szorc@gmail.com>
parents: 31878
diff changeset
    82
cookielib = pycompat.cookielib
30471
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy
Yuya Nishihara <yuya@tcha.org>
parents: 30442
diff changeset
    83
httplib = pycompat.httplib
37099
6ca5f825a0ca util: make safehasattr() a pycompat function
Yuya Nishihara <yuya@tcha.org>
parents: 37098
diff changeset
    84
safehasattr = pycompat.safehasattr
30471
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy
Yuya Nishihara <yuya@tcha.org>
parents: 30442
diff changeset
    85
socketserver = pycompat.socketserver
48873
5aafc3c5bdec py3: use io.BytesIO directly
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48870
diff changeset
    86
bytesio = io.BytesIO
36958
644a02f6b34f util: prefer "bytesio" to "stringio"
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36946
diff changeset
    87
# TODO deprecate stringio name, as it is a lie on Python 3.
644a02f6b34f util: prefer "bytesio" to "stringio"
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36946
diff changeset
    88
stringio = bytesio
30471
00c9ac4ce816 util: rewrite pycompat imports to make pyflakes always happy
Yuya Nishihara <yuya@tcha.org>
parents: 30442
diff changeset
    89
xmlrpclib = pycompat.xmlrpclib
28883
032c4c2f802a pycompat: switch to util.urlreq/util.urlerr for py3 compat
timeless <timeless@mozdev.org>
parents: 28882
diff changeset
    90
34467
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34435
diff changeset
    91
httpserver = urllibcompat.httpserver
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34435
diff changeset
    92
urlerr = urllibcompat.urlerr
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34435
diff changeset
    93
urlreq = urllibcompat.urlreq
192f7b126ed2 urllibcompat: move some adapters from pycompat to urllibcompat
Augie Fackler <augie@google.com>
parents: 34435
diff changeset
    94
32572
377c74ef008d win32mbcs: avoid unintentional failure at colorization
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 31952
diff changeset
    95
# workaround for win32mbcs
377c74ef008d win32mbcs: avoid unintentional failure at colorization
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 31952
diff changeset
    96
_filenamebytestr = pycompat.bytestr
377c74ef008d win32mbcs: avoid unintentional failure at colorization
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 31952
diff changeset
    97
34645
75979c8d4572 codemod: use pycompat.iswindows
Jun Wu <quark@fb.com>
parents: 34554
diff changeset
    98
if pycompat.iswindows:
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
    99
    from . import windows as platform
14912
ec46a7da9f2c util: move windows and posix wildcard imports to begin of file
Adrian Buehlmann <adrian@cadifra.com>
parents: 14911
diff changeset
   100
else:
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
   101
    from . import posix as platform
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   102
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
   103
_ = i18n._
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   104
47622
bb917eea1605 windows: introduce a `util.abspath` to replace os.path.abspath
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47446
diff changeset
   105
abspath = platform.abspath
29530
3239e2fdd2e2 chgserver: extract utility to bind unix domain socket to long path
Yuya Nishihara <yuya@tcha.org>
parents: 29455
diff changeset
   106
bindunixsocket = platform.bindunixsocket
14927
2aa3e07b2f07 posix, windows: introduce cachestat
Idan Kamara <idankk86@gmail.com>
parents: 14926
diff changeset
   107
cachestat = platform.cachestat
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   108
checkexec = platform.checkexec
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   109
checklink = platform.checklink
15011
5e44e4b3a0a3 util: move copymode into posix.py and windows.py
Adrian Buehlmann <adrian@cadifra.com>
parents: 15010
diff changeset
   110
copymode = platform.copymode
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   111
expandglobs = platform.expandglobs
35513
c4caf530b1c7 util: add a function to show the mount point of the filesystem
Matt Harbison <matt_harbison@yahoo.com>
parents: 35511
diff changeset
   112
getfsmountpoint = platform.getfsmountpoint
35509
beede158ea8a util: move getfstype() to the platform modules
Matt Harbison <matt_harbison@yahoo.com>
parents: 35460
diff changeset
   113
getfstype = platform.getfstype
47079
5b3513177f2b util: avoid echoing the password to the console on Windows py3 (issue6446)
Matt Harbison <matt_harbison@yahoo.com>
parents: 47062
diff changeset
   114
get_password = platform.get_password
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   115
groupmembers = platform.groupmembers
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   116
groupname = platform.groupname
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   117
isexec = platform.isexec
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   118
isowner = platform.isowner
32208
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32201
diff changeset
   119
listdir = osutil.listdir
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   120
localpath = platform.localpath
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   121
lookupreg = platform.lookupreg
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   122
makedir = platform.makedir
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   123
nlinks = platform.nlinks
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   124
normpath = platform.normpath
15488
6eff984d8e76 dirstate: fix case-folding identity for traditional Unix
Matt Mackall <mpm@selenic.com>
parents: 15392
diff changeset
   125
normcase = platform.normcase
24605
98744856b7d3 util: add normcase spec and fallback
Siddharth Agarwal <sid0@fb.com>
parents: 24439
diff changeset
   126
normcasespec = platform.normcasespec
98744856b7d3 util: add normcase spec and fallback
Siddharth Agarwal <sid0@fb.com>
parents: 24439
diff changeset
   127
normcasefallback = platform.normcasefallback
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   128
openhardlinks = platform.openhardlinks
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   129
oslink = platform.oslink
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   130
parsepatchoutput = platform.parsepatchoutput
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   131
pconvert = platform.pconvert
25420
c2ec81891502 util: add a simple poll utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25406
diff changeset
   132
poll = platform.poll
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   133
posixfile = platform.posixfile
39904
5fe0b880200e py3: convert os.readlink() path to native strings on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 39777
diff changeset
   134
readlink = platform.readlink
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   135
rename = platform.rename
24692
144883a8d0d4 util: add removedirs as platform depending function
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 24635
diff changeset
   136
removedirs = platform.removedirs
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   137
samedevice = platform.samedevice
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   138
samefile = platform.samefile
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   139
samestat = platform.samestat
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   140
setflags = platform.setflags
17560
9ee25d7b1aed util: implement a faster os.path.split for posix systems
Bryan O'Sullivan <bryano@fb.com>
parents: 17537
diff changeset
   141
split = platform.split
18026
ddc0323db78b osutil: write a C implementation of statfiles for unix
Bryan O'Sullivan <bryano@fb.com>
parents: 18013
diff changeset
   142
statfiles = getattr(osutil, 'statfiles', platform.statfiles)
18868
cafa447a7d3b util: add functions to check symlink/exec bits
Bryan O'Sullivan <bryano@fb.com>
parents: 18775
diff changeset
   143
statisexec = platform.statisexec
cafa447a7d3b util: add functions to check symlink/exec bits
Bryan O'Sullivan <bryano@fb.com>
parents: 18775
diff changeset
   144
statislink = platform.statislink
14926
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   145
umask = platform.umask
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   146
unlink = platform.unlink
4e7e63fc685a util: eliminate wildcard imports
Adrian Buehlmann <adrian@cadifra.com>
parents: 14918
diff changeset
   147
username = platform.username
14912
ec46a7da9f2c util: move windows and posix wildcard imports to begin of file
Adrian Buehlmann <adrian@cadifra.com>
parents: 14911
diff changeset
   148
44629
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   149
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
   150
def setumask(val: int) -> None:
47062
f38bf44e077f black: make codebase compatible with black v21.4b2 and v20.8b1
Kyle Lippincott <spectral@google.com>
parents: 46926
diff changeset
   151
    '''updates the umask. used by chg server'''
44629
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   152
    if pycompat.iswindows:
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   153
        return
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   154
    os.umask(val)
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   155
    global umask
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   156
    platform.umask = umask = val & 0o777
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   157
d37975386798 chgserver: update the umask cache before each run
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44452
diff changeset
   158
42041
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41834
diff changeset
   159
# small compat layer
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41834
diff changeset
   160
compengines = compression.compengines
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41834
diff changeset
   161
SERVERROLE = compression.SERVERROLE
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41834
diff changeset
   162
CLIENTROLE = compression.CLIENTROLE
3e47d1ec9da5 util: extract compression code in `mercurial.utils.compression`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 41834
diff changeset
   163
6470
ac0bcd951c2c python 2.6 compatibility: compatibility wrappers for hash functions
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 6339
diff changeset
   164
# Python compatibility
3769
96095d9ff1f8 Add encoding detection
Matt Mackall <mpm@selenic.com>
parents: 3767
diff changeset
   165
15656
4f5a78fa4917 util: clean up function ordering
Matt Mackall <mpm@selenic.com>
parents: 15611
diff changeset
   166
_notset = object()
4f5a78fa4917 util: clean up function ordering
Matt Mackall <mpm@selenic.com>
parents: 15611
diff changeset
   167
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   168
30745
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
   169
def bitsfrom(container):
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
   170
    bits = 0
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
   171
    for bit in container:
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
   172
        bits |= bit
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
   173
    return bits
c1b7b2285522 revlog: flag processor
Remi Chaintron <remi@fb.com>
parents: 30736
diff changeset
   174
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   175
31950
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
   176
# python 2.6 still have deprecation warning enabled by default. We do not want
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
   177
# to display anything to standard user so detect if we are running test and
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
   178
# only use python deprecation warning in this case.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   179
_dowarn = bool(encoding.environ.get(b'HGEMITWARNINGS'))
31950
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
   180
if _dowarn:
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
   181
    # explicitly unfilter our warning for python 2.7
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
   182
    #
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
   183
    # The option of setting PYTHONWARNINGS in the test runner was investigated.
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
   184
    # However, module name set through PYTHONWARNINGS was exactly matched, so
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
   185
    # we cannot set 'mercurial' and have it match eg: 'mercurial.scmutil'. This
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
   186
    # makes the whole PYTHONWARNINGS thing useless for our usecase.
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   187
    warnings.filterwarnings('default', '', DeprecationWarning, 'mercurial')
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   188
    warnings.filterwarnings('default', '', DeprecationWarning, 'hgext')
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   189
    warnings.filterwarnings('default', '', DeprecationWarning, 'hgext3rd')
48897
10839394f39b util: remove superfluous ispy3 test
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48875
diff changeset
   190
if _dowarn:
36588
281f66777ff0 py3: silence "bad escape" warning emitted by re.sub()
Yuya Nishihara <yuya@tcha.org>
parents: 36585
diff changeset
   191
    # silence warning emitted by passing user string to re.sub()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   192
    warnings.filterwarnings(
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   193
        'ignore', 'bad escape', DeprecationWarning, 'mercurial'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   194
    )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   195
    warnings.filterwarnings(
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   196
        'ignore', 'invalid escape sequence', DeprecationWarning, 'mercurial'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   197
    )
37455
9ecb7c471cfb py3: silence warning about deprecation of imp module
Yuya Nishihara <yuya@tcha.org>
parents: 37382
diff changeset
   198
    # TODO: reinvent imp.is_frozen()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   199
    warnings.filterwarnings(
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   200
        'ignore',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   201
        'the imp module is deprecated',
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   202
        DeprecationWarning,
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   203
        'mercurial',
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   204
    )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   205
31950
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
   206
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
   207
def nouideprecwarn(msg, version, stacklevel=1):
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
   208
    """Issue an python native deprecation warning
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
   209
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
   210
    This is a noop outside of tests, use 'ui.deprecwarn' when possible.
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
   211
    """
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
   212
    if _dowarn:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   213
        msg += (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   214
            b"\n(compatibility will be dropped after Mercurial-%s,"
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   215
            b" update your code.)"
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   216
        ) % version
36128
02ed94dd9fd6 util: call warnings.warn() with a sysstr in nouideprecwarn
Augie Fackler <augie@google.com>
parents: 36038
diff changeset
   217
        warnings.warn(pycompat.sysstr(msg), DeprecationWarning, stacklevel + 1)
44989
7af5c1f5b3a0 util: flush stderr explicitly after using warnings.warn()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44868
diff changeset
   218
        # on python 3 with chg, we will need to explicitly flush the output
7af5c1f5b3a0 util: flush stderr explicitly after using warnings.warn()
Pulkit Goyal <7895pulkit@gmail.com>
parents: 44868
diff changeset
   219
        sys.stderr.flush()
31950
cc70c6dbac30 util: add a way to issue deprecation warning without a UI object
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 31946
diff changeset
   220
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   221
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   222
DIGESTS = {
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   223
    b'md5': hashlib.md5,
44060
a61287a95dc3 core: migrate uses of hashlib.sha1 to hashutil.sha1
Augie Fackler <augie@google.com>
parents: 44035
diff changeset
   224
    b'sha1': hashutil.sha1,
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   225
    b'sha512': hashlib.sha512,
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   226
}
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   227
# List of digest types from strongest to weakest
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   228
DIGESTS_BY_STRENGTH = [b'sha512', b'sha1', b'md5']
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   229
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   230
for k in DIGESTS_BY_STRENGTH:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   231
    assert k in DIGESTS
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   232
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   233
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
   234
class digester:
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   235
    """helper to compute digests.
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   236
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   237
    This helper can be used to compute one or more digests given their name.
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   238
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
   239
    >>> d = digester([b'md5', b'sha1'])
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
   240
    >>> d.update(b'foo')
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   241
    >>> [k for k in sorted(d)]
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   242
    ['md5', 'sha1']
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
   243
    >>> d[b'md5']
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   244
    'acbd18db4cc2f85cedef654fccc4a4d8'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
   245
    >>> d[b'sha1']
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   246
    '0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33'
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
   247
    >>> digester.preferred([b'md5', b'sha1'])
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   248
    'sha1'
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   249
    """
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   250
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   251
    def __init__(self, digests, s=b''):
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   252
        self._hashes = {}
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   253
        for k in digests:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   254
            if k not in DIGESTS:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   255
                raise error.Abort(_(b'unknown digest type: %s') % k)
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   256
            self._hashes[k] = DIGESTS[k]()
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   257
        if s:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   258
            self.update(s)
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   259
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   260
    def update(self, data):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   261
        for h in self._hashes.values():
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   262
            h.update(data)
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   263
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   264
    def __getitem__(self, key):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   265
        if key not in DIGESTS:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   266
            raise error.Abort(_(b'unknown digest type: %s') % k)
46113
59fa3890d40a node: import symbols explicitly
Joerg Sonnenberger <joerg@bec.de>
parents: 45942
diff changeset
   267
        return hex(self._hashes[key].digest())
22962
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   268
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   269
    def __iter__(self):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   270
        return iter(self._hashes)
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   271
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   272
    @staticmethod
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   273
    def preferred(supported):
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   274
        """returns the strongest digest type in both supported and DIGESTS."""
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   275
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   276
        for k in DIGESTS_BY_STRENGTH:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   277
            if k in supported:
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   278
                return k
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   279
        return None
4d58f4083148 util: add a helper class to compute digests
Mike Hommey <mh@glandium.org>
parents: 22958
diff changeset
   280
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   281
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
   282
class digestchecker:
22963
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   283
    """file handle wrapper that additionally checks content against a given
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   284
    size and digests.
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   285
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   286
        d = digestchecker(fh, size, {'md5': '...'})
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   287
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   288
    When multiple digests are given, all of them are validated.
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   289
    """
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   290
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   291
    def __init__(self, fh, size, digests):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   292
        self._fh = fh
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   293
        self._size = size
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   294
        self._got = 0
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   295
        self._digests = dict(digests)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   296
        self._digester = digester(self._digests.keys())
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   297
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   298
    def read(self, length=-1):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   299
        content = self._fh.read(length)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   300
        self._digester.update(content)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   301
        self._got += len(content)
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   302
        return content
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   303
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   304
    def validate(self):
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   305
        if self._size != self._got:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   306
            raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   307
                _(b'size mismatch: expected %d, got %d')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   308
                % (self._size, self._got)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   309
            )
22963
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   310
        for k, v in self._digests.items():
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   311
            if v != self._digester[k]:
23076
c312ef382033 i18n: add hint to digest mismatch message
Wagner Bruna <wbruna@softwareexpress.com.br>
parents: 23030
diff changeset
   312
                # i18n: first parameter is a digest name
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   313
                raise error.Abort(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   314
                    _(b'%s mismatch: expected %s, got %s')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   315
                    % (k, v, self._digester[k])
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   316
                )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   317
22963
56e04741bbf1 util: add a file handle wrapper class that does hash digest validation
Mike Hommey <mh@glandium.org>
parents: 22962
diff changeset
   318
11565
7546d4a272c8 util: improved the check for the existence of the 'buffer' builtin
Renato Cunha <renatoc@gmail.com>
parents: 11469
diff changeset
   319
try:
46645
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46113
diff changeset
   320
    buffer = buffer  # pytype: disable=name-error
11565
7546d4a272c8 util: improved the check for the existence of the 'buffer' builtin
Renato Cunha <renatoc@gmail.com>
parents: 11469
diff changeset
   321
except NameError:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   322
33549
9a2ee9591acc util: remove dead code which used to be for old python2 versions
Alex Gaynor <agaynor@mozilla.com>
parents: 33446
diff changeset
   323
    def buffer(sliceable, offset=0, length=None):
9a2ee9591acc util: remove dead code which used to be for old python2 versions
Alex Gaynor <agaynor@mozilla.com>
parents: 33446
diff changeset
   324
        if length is not None:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   325
            return memoryview(sliceable)[offset : offset + length]
33549
9a2ee9591acc util: remove dead code which used to be for old python2 versions
Alex Gaynor <agaynor@mozilla.com>
parents: 33446
diff changeset
   326
        return memoryview(sliceable)[offset:]
10756
cb681cc59a8d util: fake the builtin buffer if it's missing (jython)
Ronny Pfannschmidt <Ronny.Pfannschmidt@gmx.de>
parents: 10487
diff changeset
   327
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   328
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   329
_chunksize = 4096
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   330
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   331
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
   332
class bufferedinputpipe:
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   333
    """a manually buffered input pipe
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   334
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   335
    Python will not let us use buffered IO and lazy reading with 'polling' at
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   336
    the same time. We cannot probe the buffer state and select will not detect
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   337
    that data are ready to read if they are already buffered.
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   338
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   339
    This class let us work around that by implementing its own buffering
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   340
    (allowing efficient readline) while offering a way to know if the buffer is
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   341
    empty from the output (allowing collaboration of the buffer with polling).
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   342
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   343
    This class lives in the 'util' module because it makes use of the 'os'
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   344
    module from the python stdlib.
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   345
    """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   346
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   347
    def __new__(cls, fh):
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   348
        # If we receive a fileobjectproxy, we need to use a variation of this
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   349
        # class that notifies observers about activity.
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   350
        if isinstance(fh, fileobjectproxy):
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   351
            cls = observedbufferedinputpipe
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   352
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   353
        return super(bufferedinputpipe, cls).__new__(cls)
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   354
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   355
    def __init__(self, input):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   356
        self._input = input
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   357
        self._buffer = []
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   358
        self._eof = False
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
   359
        self._lenbuf = 0
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   360
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   361
    @property
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   362
    def hasbuffer(self):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   363
        """True is any data is currently buffered
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   364
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   365
        This will be used externally a pre-step for polling IO. If there is
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   366
        already data then no polling should be set in place."""
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   367
        return bool(self._buffer)
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   368
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   369
    @property
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   370
    def closed(self):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   371
        return self._input.closed
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   372
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   373
    def fileno(self):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   374
        return self._input.fileno()
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   375
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   376
    def close(self):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   377
        return self._input.close()
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   378
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   379
    def read(self, size):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   380
        while (not self._eof) and (self._lenbuf < size):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   381
            self._fillbuffer()
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   382
        return self._frombuffer(size)
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   383
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
   384
    def unbufferedread(self, size):
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
   385
        if not self._eof and self._lenbuf == 0:
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
   386
            self._fillbuffer(max(size, _chunksize))
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
   387
        return self._frombuffer(min(self._lenbuf, size))
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
   388
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   389
    def readline(self, *args, **kwargs):
40029
e2697acd9381 cleanup: some Yoda conditions, this patch removes
Martin von Zweigbergk <martinvonz@google.com>
parents: 39905
diff changeset
   390
        if len(self._buffer) > 1:
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   391
            # this should not happen because both read and readline end with a
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   392
            # _frombuffer call that collapse it.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   393
            self._buffer = [b''.join(self._buffer)]
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
   394
            self._lenbuf = len(self._buffer[0])
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   395
        lfi = -1
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   396
        if self._buffer:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   397
            lfi = self._buffer[-1].find(b'\n')
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   398
        while (not self._eof) and lfi < 0:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   399
            self._fillbuffer()
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   400
            if self._buffer:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   401
                lfi = self._buffer[-1].find(b'\n')
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   402
        size = lfi + 1
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   403
        if lfi < 0:  # end of file
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   404
            size = self._lenbuf
40029
e2697acd9381 cleanup: some Yoda conditions, this patch removes
Martin von Zweigbergk <martinvonz@google.com>
parents: 39905
diff changeset
   405
        elif len(self._buffer) > 1:
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   406
            # we need to take previous chunks into account
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   407
            size += self._lenbuf - len(self._buffer[-1])
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   408
        return self._frombuffer(size)
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   409
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   410
    def _frombuffer(self, size):
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   411
        """return at most 'size' data from the buffer
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   412
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   413
        The data are removed from the buffer."""
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   414
        if size == 0 or not self._buffer:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   415
            return b''
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   416
        buf = self._buffer[0]
40029
e2697acd9381 cleanup: some Yoda conditions, this patch removes
Martin von Zweigbergk <martinvonz@google.com>
parents: 39905
diff changeset
   417
        if len(self._buffer) > 1:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   418
            buf = b''.join(self._buffer)
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   419
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   420
        data = buf[:size]
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   421
        buf = buf[len(data) :]
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   422
        if buf:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   423
            self._buffer = [buf]
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
   424
            self._lenbuf = len(buf)
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   425
        else:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   426
            self._buffer = []
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
   427
            self._lenbuf = 0
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   428
        return data
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   429
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
   430
    def _fillbuffer(self, size=_chunksize):
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   431
        """read data to the buffer"""
38713
27391d74aaa2 ssh: avoid reading beyond the end of stream when using compression
Joerg Sonnenberger <joerg@bec.de>
parents: 38575
diff changeset
   432
        data = os.read(self._input.fileno(), size)
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   433
        if not data:
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   434
            self._eof = True
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   435
        else:
25672
050dc6eabc92 bufferedinputpipe: remove N^2 computation of buffer length (issue4735)
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25671
diff changeset
   436
            self._lenbuf += len(data)
25406
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   437
            self._buffer.append(data)
be930f16a52a util: introduce a bufferedinputpipe utility
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 25245
diff changeset
   438
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   439
        return data
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   440
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   441
44035
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44021
diff changeset
   442
def mmapread(fp, size=None):
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44021
diff changeset
   443
    if size == 0:
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44021
diff changeset
   444
        # size of 0 to mmap.mmap() means "all data"
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44021
diff changeset
   445
        # rather than "zero bytes", so special case that.
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44021
diff changeset
   446
        return b''
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44021
diff changeset
   447
    elif size is None:
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44021
diff changeset
   448
        size = 0
48010
ae79611e3115 util: avoid a name-error warning in the `mmapread` exception handler
Matt Harbison <matt_harbison@yahoo.com>
parents: 47622
diff changeset
   449
    fd = getattr(fp, 'fileno', lambda: fp)()
34295
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34139
diff changeset
   450
    try:
44035
8ed8dfbeabb9 mmap: add a size argument to mmapread
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 44021
diff changeset
   451
        return mmap.mmap(fd, size, access=mmap.ACCESS_READ)
34295
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34139
diff changeset
   452
    except ValueError:
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34139
diff changeset
   453
        # Empty files cannot be mmapped, but mmapread should still work.  Check
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34139
diff changeset
   454
        # if the file is empty, and if so, return an empty buffer.
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34139
diff changeset
   455
        if os.fstat(fd).st_size == 0:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   456
            return b''
34295
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34139
diff changeset
   457
        raise
3bb2a9f25fe9 util: add an mmapread method
Mark Thomas <mbthomas@fb.com>
parents: 34139
diff changeset
   458
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   459
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
   460
class fileobjectproxy:
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   461
    """A proxy around file objects that tells a watcher when events occur.
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   462
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   463
    This type is intended to only be used for testing purposes. Think hard
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   464
    before using it in important code.
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   465
    """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   466
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   467
    __slots__ = (
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   468
        '_orig',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   469
        '_observer',
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   470
    )
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   471
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   472
    def __init__(self, fh, observer):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   473
        object.__setattr__(self, '_orig', fh)
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   474
        object.__setattr__(self, '_observer', observer)
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   475
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   476
    def __getattribute__(self, name):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   477
        ours = {
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   478
            '_observer',
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   479
            # IOBase
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   480
            'close',
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   481
            # closed if a property
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   482
            'fileno',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   483
            'flush',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   484
            'isatty',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   485
            'readable',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   486
            'readline',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   487
            'readlines',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   488
            'seek',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   489
            'seekable',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   490
            'tell',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   491
            'truncate',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   492
            'writable',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   493
            'writelines',
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   494
            # RawIOBase
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   495
            'read',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   496
            'readall',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   497
            'readinto',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   498
            'write',
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   499
            # BufferedIOBase
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   500
            # raw is a property
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   501
            'detach',
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   502
            # read defined above
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   503
            'read1',
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   504
            # readinto defined above
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   505
            # write defined above
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   506
        }
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   507
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   508
        # We only observe some methods.
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   509
        if name in ours:
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   510
            return object.__getattribute__(self, name)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   511
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   512
        return getattr(object.__getattribute__(self, '_orig'), name)
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   513
36832
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy
Matt Harbison <matt_harbison@yahoo.com>
parents: 36793
diff changeset
   514
    def __nonzero__(self):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   515
        return bool(object.__getattribute__(self, '_orig'))
36832
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy
Matt Harbison <matt_harbison@yahoo.com>
parents: 36793
diff changeset
   516
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy
Matt Harbison <matt_harbison@yahoo.com>
parents: 36793
diff changeset
   517
    __bool__ = __nonzero__
6bdea0efdab5 util: forward __bool__()/__nonzero__() on fileobjectproxy
Matt Harbison <matt_harbison@yahoo.com>
parents: 36793
diff changeset
   518
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   519
    def __delattr__(self, name):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   520
        return delattr(object.__getattribute__(self, '_orig'), name)
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   521
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   522
    def __setattr__(self, name, value):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   523
        return setattr(object.__getattribute__(self, '_orig'), name, value)
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   525
    def __iter__(self):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   526
        return object.__getattribute__(self, '_orig').__iter__()
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   527
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   528
    def _observedcall(self, name, *args, **kwargs):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   529
        # Call the original object.
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   530
        orig = object.__getattribute__(self, '_orig')
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   531
        res = getattr(orig, name)(*args, **kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   532
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   533
        # Call a method on the observer of the same name with arguments
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   534
        # so it can react, log, etc.
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   535
        observer = object.__getattribute__(self, '_observer')
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   536
        fn = getattr(observer, name, None)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   537
        if fn:
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   538
            fn(res, *args, **kwargs)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   539
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   540
        return res
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   541
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   542
    def close(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   543
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   544
            'close', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   545
        )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   546
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   547
    def fileno(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   548
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   549
            'fileno', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   550
        )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   551
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   552
    def flush(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   553
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   554
            'flush', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   555
        )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   556
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   557
    def isatty(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   558
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   559
            'isatty', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   560
        )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   561
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   562
    def readable(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   563
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   564
            'readable', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   565
        )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   566
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   567
    def readline(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   568
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   569
            'readline', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   570
        )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   571
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   572
    def readlines(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   573
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   574
            'readlines', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   575
        )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   576
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   577
    def seek(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   578
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   579
            'seek', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   580
        )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   581
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   582
    def seekable(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   583
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   584
            'seekable', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   585
        )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   586
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   587
    def tell(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   588
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   589
            'tell', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   590
        )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   591
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   592
    def truncate(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   593
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   594
            'truncate', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   595
        )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   596
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   597
    def writable(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   598
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   599
            'writable', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   600
        )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   601
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   602
    def writelines(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   603
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   604
            'writelines', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   605
        )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   606
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   607
    def read(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   608
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   609
            'read', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   610
        )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   611
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   612
    def readall(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   613
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   614
            'readall', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   615
        )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   616
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   617
    def readinto(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   618
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   619
            'readinto', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   620
        )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   621
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   622
    def write(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   623
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   624
            'write', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   625
        )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   626
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   627
    def detach(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   628
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   629
            'detach', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   630
        )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   631
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   632
    def read1(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   633
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   634
            'read1', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   635
        )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   636
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   637
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   638
class observedbufferedinputpipe(bufferedinputpipe):
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   639
    """A variation of bufferedinputpipe that is aware of fileobjectproxy.
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   640
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   641
    ``bufferedinputpipe`` makes low-level calls to ``os.read()`` that
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   642
    bypass ``fileobjectproxy``. Because of this, we need to make
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   643
    ``bufferedinputpipe`` aware of these operations.
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   644
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   645
    This variation of ``bufferedinputpipe`` can notify observers about
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   646
    ``os.read()`` events. It also re-publishes other events, such as
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   647
    ``read()`` and ``readline()``.
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   648
    """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   649
49814
1d1b244a91b6 util: fix the signature of observedbufferedinputpipe._fillbuffer()
Matt Harbison <matt_harbison@yahoo.com>
parents: 49813
diff changeset
   650
    def _fillbuffer(self, size=_chunksize):
1d1b244a91b6 util: fix the signature of observedbufferedinputpipe._fillbuffer()
Matt Harbison <matt_harbison@yahoo.com>
parents: 49813
diff changeset
   651
        res = super(observedbufferedinputpipe, self)._fillbuffer(size=size)
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   652
43103
c95b2f40db7c py3: stop normalizing 2nd argument of *attr() to unicode
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43090
diff changeset
   653
        fn = getattr(self._input._observer, 'osread', None)
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   654
        if fn:
49814
1d1b244a91b6 util: fix the signature of observedbufferedinputpipe._fillbuffer()
Matt Harbison <matt_harbison@yahoo.com>
parents: 49813
diff changeset
   655
            fn(res, size)
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   656
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   657
        return res
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   658
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   659
    # We use different observer methods because the operation isn't
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   660
    # performed on the actual file object but on us.
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   661
    def read(self, size):
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   662
        res = super(observedbufferedinputpipe, self).read(size)
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   663
43103
c95b2f40db7c py3: stop normalizing 2nd argument of *attr() to unicode
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43090
diff changeset
   664
        fn = getattr(self._input._observer, 'bufferedread', None)
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   665
        if fn:
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   666
            fn(res, size)
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   667
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   668
        return res
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   669
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   670
    def readline(self, *args, **kwargs):
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   671
        res = super(observedbufferedinputpipe, self).readline(*args, **kwargs)
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   672
43103
c95b2f40db7c py3: stop normalizing 2nd argument of *attr() to unicode
Gregory Szorc <gregory.szorc@gmail.com>
parents: 43090
diff changeset
   673
        fn = getattr(self._input._observer, 'bufferedreadline', None)
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   674
        if fn:
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   675
            fn(res)
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   676
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   677
        return res
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   678
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   679
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   680
PROXIED_SOCKET_METHODS = {
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   681
    'makefile',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   682
    'recv',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   683
    'recvfrom',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   684
    'recvfrom_into',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   685
    'recv_into',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   686
    'send',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   687
    'sendall',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   688
    'sendto',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   689
    'setblocking',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   690
    'settimeout',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   691
    'gettimeout',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   692
    'setsockopt',
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   693
}
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   694
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   695
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
   696
class socketproxy:
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   697
    """A proxy around a socket that tells a watcher when events occur.
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   698
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   699
    This is like ``fileobjectproxy`` except for sockets.
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   700
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   701
    This type is intended to only be used for testing purposes. Think hard
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   702
    before using it in important code.
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   703
    """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   704
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   705
    __slots__ = (
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   706
        '_orig',
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   707
        '_observer',
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   708
    )
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   709
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   710
    def __init__(self, sock, observer):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   711
        object.__setattr__(self, '_orig', sock)
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   712
        object.__setattr__(self, '_observer', observer)
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   713
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   714
    def __getattribute__(self, name):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   715
        if name in PROXIED_SOCKET_METHODS:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   716
            return object.__getattribute__(self, name)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   717
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   718
        return getattr(object.__getattribute__(self, '_orig'), name)
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   719
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   720
    def __delattr__(self, name):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   721
        return delattr(object.__getattribute__(self, '_orig'), name)
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   722
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   723
    def __setattr__(self, name, value):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   724
        return setattr(object.__getattribute__(self, '_orig'), name, value)
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   725
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   726
    def __nonzero__(self):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   727
        return bool(object.__getattribute__(self, '_orig'))
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   728
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   729
    __bool__ = __nonzero__
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   730
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   731
    def _observedcall(self, name, *args, **kwargs):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   732
        # Call the original object.
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   733
        orig = object.__getattribute__(self, '_orig')
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   734
        res = getattr(orig, name)(*args, **kwargs)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   735
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   736
        # Call a method on the observer of the same name with arguments
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   737
        # so it can react, log, etc.
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   738
        observer = object.__getattribute__(self, '_observer')
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   739
        fn = getattr(observer, name, None)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   740
        if fn:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   741
            fn(res, *args, **kwargs)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   742
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   743
        return res
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   744
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   745
    def makefile(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   746
        res = object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   747
            'makefile', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   748
        )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   749
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   750
        # The file object may be used for I/O. So we turn it into a
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   751
        # proxy using our observer.
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   752
        observer = object.__getattribute__(self, '_observer')
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   753
        return makeloggingfileobject(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   754
            observer.fh,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   755
            res,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   756
            observer.name,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   757
            reads=observer.reads,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   758
            writes=observer.writes,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   759
            logdata=observer.logdata,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   760
            logdataapis=observer.logdataapis,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   761
        )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   762
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   763
    def recv(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   764
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   765
            'recv', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   766
        )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   767
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   768
    def recvfrom(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   769
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   770
            'recvfrom', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   771
        )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   772
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   773
    def recvfrom_into(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   774
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   775
            'recvfrom_into', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   776
        )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   777
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   778
    def recv_into(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   779
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   780
            'recv_info', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   781
        )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   782
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   783
    def send(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   784
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   785
            'send', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   786
        )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   787
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   788
    def sendall(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   789
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   790
            'sendall', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   791
        )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   792
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   793
    def sendto(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   794
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   795
            'sendto', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   796
        )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   797
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   798
    def setblocking(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   799
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   800
            'setblocking', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   801
        )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   802
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   803
    def settimeout(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   804
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   805
            'settimeout', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   806
        )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   807
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   808
    def gettimeout(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   809
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   810
            'gettimeout', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   811
        )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   812
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   813
    def setsockopt(self, *args, **kwargs):
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   814
        return object.__getattribute__(self, '_observedcall')(
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
   815
            'setsockopt', *args, **kwargs
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   816
        )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   817
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   818
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
   819
class baseproxyobserver:
43920
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43907
diff changeset
   820
    def __init__(self, fh, name, logdata, logdataapis):
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43907
diff changeset
   821
        self.fh = fh
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43907
diff changeset
   822
        self.name = name
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43907
diff changeset
   823
        self.logdata = logdata
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43907
diff changeset
   824
        self.logdataapis = logdataapis
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43907
diff changeset
   825
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   826
    def _writedata(self, data):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   827
        if not self.logdata:
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   828
            if self.logdataapis:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   829
                self.fh.write(b'\n')
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   830
                self.fh.flush()
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   831
            return
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   832
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   833
        # Simple case writes all data on a single line.
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   834
        if b'\n' not in data:
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   835
            if self.logdataapis:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   836
                self.fh.write(b': %s\n' % stringutil.escapestr(data))
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   837
            else:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   838
                self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   839
                    b'%s>     %s\n' % (self.name, stringutil.escapestr(data))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   840
                )
36991
d683c7367989 wireproto: explicitly flush stdio to prevent stalls on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 36958
diff changeset
   841
            self.fh.flush()
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   842
            return
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   843
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   844
        # Data with newlines is written to multiple lines.
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   845
        if self.logdataapis:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   846
            self.fh.write(b':\n')
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   847
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   848
        lines = data.splitlines(True)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   849
        for line in lines:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   850
            self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   851
                b'%s>     %s\n' % (self.name, stringutil.escapestr(line))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   852
            )
36991
d683c7367989 wireproto: explicitly flush stdio to prevent stalls on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 36958
diff changeset
   853
        self.fh.flush()
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   854
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   855
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   856
class fileobjectobserver(baseproxyobserver):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   857
    """Logs file object activity."""
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   858
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   859
    def __init__(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   860
        self, fh, name, reads=True, writes=True, logdata=False, logdataapis=True
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   861
    ):
43920
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43907
diff changeset
   862
        super(fileobjectobserver, self).__init__(fh, name, logdata, logdataapis)
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   863
        self.reads = reads
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   864
        self.writes = writes
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   865
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   866
    def read(self, res, size=-1):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   867
        if not self.reads:
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   868
            return
36585
26a6b62919e2 util: work around Python 3 returning None at EOF instead of ''
Augie Fackler <augie@google.com>
parents: 36584
diff changeset
   869
        # Python 3 can return None from reads at EOF instead of empty strings.
26a6b62919e2 util: work around Python 3 returning None at EOF instead of ''
Augie Fackler <augie@google.com>
parents: 36584
diff changeset
   870
        if res is None:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   871
            res = b''
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   872
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   873
        if size == -1 and res == b'':
38314
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver
Augie Fackler <augie@google.com>
parents: 38164
diff changeset
   874
            # Suppress pointless read(-1) calls that return
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver
Augie Fackler <augie@google.com>
parents: 38164
diff changeset
   875
            # nothing. These happen _a lot_ on Python 3, and there
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver
Augie Fackler <augie@google.com>
parents: 38164
diff changeset
   876
            # doesn't seem to be a better workaround to have matching
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver
Augie Fackler <augie@google.com>
parents: 38164
diff changeset
   877
            # Python 2 and 3 behavior. :(
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver
Augie Fackler <augie@google.com>
parents: 38164
diff changeset
   878
            return
565074cc9ac6 tests: suppress read(-1) -> '' calls in fileobjectobserver
Augie Fackler <augie@google.com>
parents: 38164
diff changeset
   879
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   880
        if self.logdataapis:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   881
            self.fh.write(b'%s> read(%d) -> %d' % (self.name, size, len(res)))
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   882
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   883
        self._writedata(res)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   884
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   885
    def readline(self, res, limit=-1):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   886
        if not self.reads:
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   887
            return
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   888
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   889
        if self.logdataapis:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   890
            self.fh.write(b'%s> readline() -> %d' % (self.name, len(res)))
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   891
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   892
        self._writedata(res)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   893
36630
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36629
diff changeset
   894
    def readinto(self, res, dest):
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36629
diff changeset
   895
        if not self.reads:
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36629
diff changeset
   896
            return
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36629
diff changeset
   897
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   898
        if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   899
            self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   900
                b'%s> readinto(%d) -> %r' % (self.name, len(dest), res)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   901
            )
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   902
36630
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36629
diff changeset
   903
        data = dest[0:res] if res is not None else b''
41384
b141b5243b37 util: cast memoryview to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 41289
diff changeset
   904
b141b5243b37 util: cast memoryview to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 41289
diff changeset
   905
        # _writedata() uses "in" operator and is confused by memoryview because
b141b5243b37 util: cast memoryview to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 41289
diff changeset
   906
        # characters are ints on Python 3.
b141b5243b37 util: cast memoryview to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 41289
diff changeset
   907
        if isinstance(data, memoryview):
b141b5243b37 util: cast memoryview to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 41289
diff changeset
   908
            data = data.tobytes()
b141b5243b37 util: cast memoryview to bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 41289
diff changeset
   909
36630
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36629
diff changeset
   910
        self._writedata(data)
29128309c52d util: log readinto() I/O
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36629
diff changeset
   911
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   912
    def write(self, res, data):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   913
        if not self.writes:
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   914
            return
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   915
36631
8395fddde46c util: report integer result from write()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36630
diff changeset
   916
        # Python 2 returns None from some write() calls. Python 3 (reasonably)
8395fddde46c util: report integer result from write()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36630
diff changeset
   917
        # returns the integer bytes written.
8395fddde46c util: report integer result from write()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36630
diff changeset
   918
        if res is None and data:
8395fddde46c util: report integer result from write()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36630
diff changeset
   919
            res = len(data)
8395fddde46c util: report integer result from write()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36630
diff changeset
   920
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   921
        if self.logdataapis:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   922
            self.fh.write(b'%s> write(%d) -> %r' % (self.name, len(data), res))
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   923
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   924
        self._writedata(data)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   925
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   926
    def flush(self, res):
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   927
        if not self.writes:
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   928
            return
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   929
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   930
        self.fh.write(b'%s> flush() -> %r\n' % (self.name, res))
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   931
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   932
    # For observedbufferedinputpipe.
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   933
    def bufferedread(self, res, size):
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   934
        if not self.reads:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   935
            return
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   936
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   937
        if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   938
            self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   939
                b'%s> bufferedread(%d) -> %d' % (self.name, size, len(res))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   940
            )
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   941
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   942
        self._writedata(res)
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   943
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   944
    def bufferedreadline(self, res):
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   945
        if not self.reads:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   946
            return
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   947
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   948
        if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   949
            self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   950
                b'%s> bufferedreadline() -> %d' % (self.name, len(res))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   951
            )
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
   952
36525
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   953
        self._writedata(res)
3158052720ae util: enable observing of util.bufferedinputpipe
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36524
diff changeset
   954
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   955
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   956
def makeloggingfileobject(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   957
    logh, fh, name, reads=True, writes=True, logdata=False, logdataapis=True
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   958
):
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   959
    """Turn a file object into a logging file object."""
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   960
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   961
    observer = fileobjectobserver(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   962
        logh,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   963
        name,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   964
        reads=reads,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   965
        writes=writes,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   966
        logdata=logdata,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   967
        logdataapis=logdataapis,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   968
    )
36524
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   969
    return fileobjectproxy(fh, observer)
bfe38f787d5b util: add a file object proxy that can notify observers
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36432
diff changeset
   970
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   971
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   972
class socketobserver(baseproxyobserver):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   973
    """Logs socket activity."""
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   974
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   975
    def __init__(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   976
        self,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   977
        fh,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   978
        name,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   979
        reads=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   980
        writes=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   981
        states=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   982
        logdata=False,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   983
        logdataapis=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
   984
    ):
43920
4222b9d5d4fb util: move common proxyobserver attributes to the base class
Matt Harbison <matt_harbison@yahoo.com>
parents: 43907
diff changeset
   985
        super(socketobserver, self).__init__(fh, name, logdata, logdataapis)
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   986
        self.reads = reads
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   987
        self.writes = writes
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   988
        self.states = states
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   989
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   990
    def makefile(self, res, mode=None, bufsize=None):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   991
        if not self.states:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   992
            return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   993
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
   994
        self.fh.write(b'%s> makefile(%r, %r)\n' % (self.name, mode, bufsize))
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   995
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   996
    def recv(self, res, size, flags=0):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   997
        if not self.reads:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   998
            return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
   999
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
  1000
        if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1001
            self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1002
                b'%s> recv(%d, %d) -> %d' % (self.name, size, flags, len(res))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1003
            )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1004
        self._writedata(res)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1005
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1006
    def recvfrom(self, res, size, flags=0):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1007
        if not self.reads:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1008
            return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1009
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
  1010
        if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1011
            self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1012
                b'%s> recvfrom(%d, %d) -> %d'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1013
                % (self.name, size, flags, len(res[0]))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1014
            )
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
  1015
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1016
        self._writedata(res[0])
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1017
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1018
    def recvfrom_into(self, res, buf, size, flags=0):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1019
        if not self.reads:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1020
            return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1021
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
  1022
        if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1023
            self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1024
                b'%s> recvfrom_into(%d, %d) -> %d'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1025
                % (self.name, size, flags, res[0])
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1026
            )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1027
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1028
        self._writedata(buf[0 : res[0]])
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1029
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1030
    def recv_into(self, res, buf, size=0, flags=0):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1031
        if not self.reads:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1032
            return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1033
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
  1034
        if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1035
            self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1036
                b'%s> recv_into(%d, %d) -> %d' % (self.name, size, flags, res)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1037
            )
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
  1038
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1039
        self._writedata(buf[0:res])
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1040
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1041
    def send(self, res, data, flags=0):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1042
        if not self.writes:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1043
            return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1044
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1045
        self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1046
            b'%s> send(%d, %d) -> %d' % (self.name, len(data), flags, len(res))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1047
        )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1048
        self._writedata(data)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1049
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1050
    def sendall(self, res, data, flags=0):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1051
        if not self.writes:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1052
            return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1053
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
  1054
        if self.logdataapis:
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
  1055
            # Returns None on success. So don't bother reporting return value.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1056
            self.fh.write(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1057
                b'%s> sendall(%d, %d)' % (self.name, len(data), flags)
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1058
            )
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
  1059
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1060
        self._writedata(data)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1061
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1062
    def sendto(self, res, data, flagsoraddress, address=None):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1063
        if not self.writes:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1064
            return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1065
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1066
        if address:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1067
            flags = flagsoraddress
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1068
        else:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1069
            flags = 0
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1070
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
  1071
        if self.logdataapis:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1072
            self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1073
                b'%s> sendto(%d, %d, %r) -> %d'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1074
                % (self.name, len(data), flags, address, res)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1075
            )
37044
d3a9036d9ae9 util: don't log low-level I/O calls for HTTP peer
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37010
diff changeset
  1076
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1077
        self._writedata(data)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1078
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1079
    def setblocking(self, res, flag):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1080
        if not self.states:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1081
            return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1082
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1083
        self.fh.write(b'%s> setblocking(%r)\n' % (self.name, flag))
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1084
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1085
    def settimeout(self, res, value):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1086
        if not self.states:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1087
            return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1088
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1089
        self.fh.write(b'%s> settimeout(%r)\n' % (self.name, value))
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1090
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1091
    def gettimeout(self, res):
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1092
        if not self.states:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1093
            return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1094
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1095
        self.fh.write(b'%s> gettimeout() -> %f\n' % (self.name, res))
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1096
39060
ed8160e4fea0 util: fix signature of setsockopt in socket observer
Augie Fackler <augie@google.com>
parents: 38812
diff changeset
  1097
    def setsockopt(self, res, level, optname, value):
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1098
        if not self.states:
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1099
            return
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1100
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1101
        self.fh.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1102
            b'%s> setsockopt(%r, %r, %r) -> %r\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1103
            % (self.name, level, optname, value, res)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1104
        )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1105
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1106
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1107
def makeloggingsocket(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1108
    logh,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1109
    fh,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1110
    name,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1111
    reads=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1112
    writes=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1113
    states=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1114
    logdata=False,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1115
    logdataapis=True,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1116
):
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1117
    """Turn a socket into a logging socket."""
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1118
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1119
    observer = socketobserver(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1120
        logh,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1121
        name,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1122
        reads=reads,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1123
        writes=writes,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1124
        states=states,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1125
        logdata=logdata,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1126
        logdataapis=logdataapis,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1127
    )
37010
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1128
    return socketproxy(fh, observer)
8453699a1f21 util: observable proxy objects for sockets
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36991
diff changeset
  1129
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1130
7632
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
  1131
def version():
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
  1132
    """Return version information if available."""
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
  1133
    try:
27358
ac839ee45b6a util: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27357
diff changeset
  1134
        from . import __version__
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1135
7632
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
  1136
        return __version__.version
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
  1137
    except ImportError:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1138
        return b'unknown'
7632
9626819b2e3d refactor version code
Matt Mackall <mpm@selenic.com>
parents: 7559
diff changeset
  1139
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1140
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1141
def versiontuple(v=None, n=4):
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1142
    """Parses a Mercurial version string into an N-tuple.
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1143
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1144
    The version string to be parsed is specified with the ``v`` argument.
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1145
    If it isn't defined, the current Mercurial version string will be parsed.
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1146
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1147
    ``n`` can be 2, 3, or 4. Here is how some version strings map to
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1148
    returned values:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1149
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  1150
    >>> v = b'3.6.1+190-df9b73d2d444'
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1151
    >>> versiontuple(v, 2)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1152
    (3, 6)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1153
    >>> versiontuple(v, 3)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1154
    (3, 6, 1)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1155
    >>> versiontuple(v, 4)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1156
    (3, 6, 1, '190-df9b73d2d444')
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1157
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  1158
    >>> versiontuple(b'3.6.1+190-df9b73d2d444+20151118')
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1159
    (3, 6, 1, '190-df9b73d2d444+20151118')
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1160
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  1161
    >>> v = b'3.6'
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1162
    >>> versiontuple(v, 2)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1163
    (3, 6)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1164
    >>> versiontuple(v, 3)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1165
    (3, 6, None)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1166
    >>> versiontuple(v, 4)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1167
    (3, 6, None, None)
29613
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
  1168
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  1169
    >>> v = b'3.9-rc'
29613
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
  1170
    >>> versiontuple(v, 2)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
  1171
    (3, 9)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
  1172
    >>> versiontuple(v, 3)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
  1173
    (3, 9, None)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
  1174
    >>> versiontuple(v, 4)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
  1175
    (3, 9, None, 'rc')
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
  1176
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  1177
    >>> v = b'3.9-rc+2-02a8fea4289b'
29613
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
  1178
    >>> versiontuple(v, 2)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
  1179
    (3, 9)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
  1180
    >>> versiontuple(v, 3)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
  1181
    (3, 9, None)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
  1182
    >>> versiontuple(v, 4)
616cbcb59e05 util: better handle '-' in version string (issue5302)
Gregory Szorc <gregory.szorc@gmail.com>
parents: 29592
diff changeset
  1183
    (3, 9, None, 'rc+2-02a8fea4289b')
37801
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
  1184
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
  1185
    >>> versiontuple(b'4.6rc0')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
  1186
    (4, 6, None, 'rc0')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
  1187
    >>> versiontuple(b'4.6rc0+12-425d55e54f98')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
  1188
    (4, 6, None, 'rc0+12-425d55e54f98')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
  1189
    >>> versiontuple(b'.1.2.3')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
  1190
    (None, None, None, '.1.2.3')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
  1191
    >>> versiontuple(b'12.34..5')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
  1192
    (12, 34, None, '..5')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
  1193
    >>> versiontuple(b'1.2.3.4.5.6')
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
  1194
    (1, 2, 3, '.4.5.6')
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1195
    """
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1196
    if not v:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1197
        v = version()
44021
6d3b67a837a6 cleanup: drop redundant character escapes from `[]` character sets
Matt Harbison <matt_harbison@yahoo.com>
parents: 44018
diff changeset
  1198
    m = remod.match(br'(\d+(?:\.\d+){,2})[+-]?(.*)', v)
37801
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
  1199
    if not m:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1200
        vparts, extra = b'', v
37801
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
  1201
    elif m.group(2):
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
  1202
        vparts, extra = m.groups()
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1203
    else:
37801
5cab6f5016fa version: make parser more robust for rc variants and ill-formed strings
Yuya Nishihara <yuya@tcha.org>
parents: 37675
diff changeset
  1204
        vparts, extra = m.group(1), None
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1205
43882
a21a6dad4b38 typing: add an assertion to util.versiontuple
Matt Harbison <matt_harbison@yahoo.com>
parents: 43881
diff changeset
  1206
    assert vparts is not None  # help pytype
a21a6dad4b38 typing: add an assertion to util.versiontuple
Matt Harbison <matt_harbison@yahoo.com>
parents: 43881
diff changeset
  1207
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1208
    vints = []
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1209
    for i in vparts.split(b'.'):
27112
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1210
        try:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1211
            vints.append(int(i))
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1212
        except ValueError:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1213
            break
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1214
    # (3, 6) -> (3, 6, None)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1215
    while len(vints) < 3:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1216
        vints.append(None)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1217
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1218
    if n == 2:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1219
        return (vints[0], vints[1])
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1220
    if n == 3:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1221
        return (vints[0], vints[1], vints[2])
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1222
    if n == 4:
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1223
        return (vints[0], vints[1], vints[2], extra)
39c14e89b881 util: add versiontuple() for returning parsed version information
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27066
diff changeset
  1224
48013
376d08ae904f util: eliminate the possibility of returning None from `versiontuple()`
Matt Harbison <matt_harbison@yahoo.com>
parents: 48010
diff changeset
  1225
    raise error.ProgrammingError(b"invalid version part request: %d" % n)
376d08ae904f util: eliminate the possibility of returning None from `versiontuple()`
Matt Harbison <matt_harbison@yahoo.com>
parents: 48010
diff changeset
  1226
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1227
3145
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
  1228
def cachefunc(func):
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
  1229
    '''cache the result of function calls'''
3147
97420a49188d add comments in cachefunc
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3145
diff changeset
  1230
    # XXX doesn't handle keywords args
28832
f5ff10f6fa6b util: use __code__ (available since py2.6)
timeless <timeless@mozdev.org>
parents: 28826
diff changeset
  1231
    if func.__code__.co_argcount == 0:
43880
eff050dbb703 util: rename a variable to avoid confusing pytype
Matt Harbison <matt_harbison@yahoo.com>
parents: 43879
diff changeset
  1232
        listcache = []
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1233
20835
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20679
diff changeset
  1234
        def f():
43880
eff050dbb703 util: rename a variable to avoid confusing pytype
Matt Harbison <matt_harbison@yahoo.com>
parents: 43879
diff changeset
  1235
            if len(listcache) == 0:
eff050dbb703 util: rename a variable to avoid confusing pytype
Matt Harbison <matt_harbison@yahoo.com>
parents: 43879
diff changeset
  1236
                listcache.append(func())
eff050dbb703 util: rename a variable to avoid confusing pytype
Matt Harbison <matt_harbison@yahoo.com>
parents: 43879
diff changeset
  1237
            return listcache[0]
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1238
20835
0e8417131a29 util: add the code path to "cachefunc()" for the function taking no arguments
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 20679
diff changeset
  1239
        return f
3145
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
  1240
    cache = {}
28832
f5ff10f6fa6b util: use __code__ (available since py2.6)
timeless <timeless@mozdev.org>
parents: 28826
diff changeset
  1241
    if func.__code__.co_argcount == 1:
3147
97420a49188d add comments in cachefunc
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3145
diff changeset
  1242
        # we gain a small amount of time because
97420a49188d add comments in cachefunc
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3145
diff changeset
  1243
        # we don't need to pack/unpack the list
3145
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
  1244
        def f(arg):
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
  1245
            if arg not in cache:
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
  1246
                cache[arg] = func(arg)
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
  1247
            return cache[arg]
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1248
3145
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
  1249
    else:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1250
3145
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
  1251
        def f(*args):
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
  1252
            if args not in cache:
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
  1253
                cache[args] = func(*args)
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
  1254
            return cache[args]
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
  1255
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
  1256
    return f
e4ea47c21480 Add cachefunc to abstract function call cache
Brendan Cully <brendan@kublai.com>
parents: 3131
diff changeset
  1257
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1258
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
  1259
class cow:
34357
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1260
    """helper class to make copy-on-write easier
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1261
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1262
    Call preparewrite before doing any writes.
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1263
    """
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1264
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1265
    def preparewrite(self):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1266
        """call this before writes, return self or a copied new object"""
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1267
        if getattr(self, '_copied', 0):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1268
            self._copied -= 1
46901
51841b23670b typing: make minor adjustments to mercurial/util.py to pass pytype checking
Matt Harbison <matt_harbison@yahoo.com>
parents: 46900
diff changeset
  1269
            # Function cow.__init__ expects 1 arg(s), got 2 [wrong-arg-count]
51841b23670b typing: make minor adjustments to mercurial/util.py to pass pytype checking
Matt Harbison <matt_harbison@yahoo.com>
parents: 46900
diff changeset
  1270
            return self.__class__(self)  # pytype: disable=wrong-arg-count
34357
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1271
        return self
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1272
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1273
    def copy(self):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1274
        """always do a cheap copy"""
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1275
        self._copied = getattr(self, '_copied', 0) + 1
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1276
        return self
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1277
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1278
32300
bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict
Martin von Zweigbergk <martinvonz@google.com>
parents: 32291
diff changeset
  1279
class sortdict(collections.OrderedDict):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  1280
    """a simple sorted dictionary
32306
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32300
diff changeset
  1281
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  1282
    >>> d1 = sortdict([(b'a', 0), (b'b', 1)])
32306
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32300
diff changeset
  1283
    >>> d2 = d1.copy()
50753
a2df74853f8d tests: fix sortdict doctest with Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50689
diff changeset
  1284
    >>> list(d2.items())
a2df74853f8d tests: fix sortdict doctest with Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50689
diff changeset
  1285
    [('a', 0), ('b', 1)]
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  1286
    >>> d2.update([(b'a', 2)])
34139
be00af4a1ac5 doctest: coerce dict.keys() to list
Yuya Nishihara <yuya@tcha.org>
parents: 34137
diff changeset
  1287
    >>> list(d2.keys()) # should still be in last-set order
32306
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32300
diff changeset
  1288
    ['b', 'a']
43907
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
  1289
    >>> d1.insert(1, b'a.5', 0.5)
50753
a2df74853f8d tests: fix sortdict doctest with Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50689
diff changeset
  1290
    >>> list(d1.items())
a2df74853f8d tests: fix sortdict doctest with Python 3.12
Mads Kiilerich <mads@kiilerich.com>
parents: 50689
diff changeset
  1291
    [('a', 0), ('a.5', 0.5), ('b', 1)]
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  1292
    """
32306
2d19664e257d util: drop unneeded override, sortdict.copy()
Yuya Nishihara <yuya@tcha.org>
parents: 32300
diff changeset
  1293
32300
bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict
Martin von Zweigbergk <martinvonz@google.com>
parents: 32291
diff changeset
  1294
    def __setitem__(self, key, value):
21813
c2262004c2e2 config: move config.sortdict class into util
Angel Ezquerra <angel.ezquerra@gmail.com>
parents: 21046
diff changeset
  1295
        if key in self:
32300
bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict
Martin von Zweigbergk <martinvonz@google.com>
parents: 32291
diff changeset
  1296
            del self[key]
bd0fd3ff9916 util: rewrite sortdict using Python 2.7's OrderedDict
Martin von Zweigbergk <martinvonz@google.com>
parents: 32291
diff changeset
  1297
        super(sortdict, self).__setitem__(key, value)
21813
c2262004c2e2 config: move config.sortdict class into util
Angel Ezquerra <angel.ezquerra@gmail.com>
parents: 21046
diff changeset
  1298
33626
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
  1299
    if pycompat.ispypy:
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
  1300
        # __setitem__() isn't called as of PyPy 5.8.0
46900
64400d05db1e util: fix the signature for the pypy override of sortdict.update()
Matt Harbison <matt_harbison@yahoo.com>
parents: 46827
diff changeset
  1301
        def update(self, src, **f):
33626
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
  1302
            if isinstance(src, dict):
48913
f254fc73d956 global: bulk replace simple pycompat.iteritems(x) with x.items()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48897
diff changeset
  1303
                src = src.items()
33626
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
  1304
            for k, v in src:
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
  1305
                self[k] = v
46900
64400d05db1e util: fix the signature for the pypy override of sortdict.update()
Matt Harbison <matt_harbison@yahoo.com>
parents: 46827
diff changeset
  1306
            for k in f:
64400d05db1e util: fix the signature for the pypy override of sortdict.update()
Matt Harbison <matt_harbison@yahoo.com>
parents: 46827
diff changeset
  1307
                self[k] = f[k]
33626
524b13fc711f util: fix sortdict.update() to call __setitem__() on PyPy (issue5639)
Yuya Nishihara <yuya@tcha.org>
parents: 33549
diff changeset
  1308
43907
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
  1309
    def insert(self, position, key, value):
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
  1310
        for (i, (k, v)) in enumerate(list(self.items())):
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
  1311
            if i == position:
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
  1312
                self[key] = value
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
  1313
            if i >= position:
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
  1314
                del self[k]
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
  1315
                self[k] = v
68af0228fedc util: implement sortdict.insert()
Martin von Zweigbergk <martinvonz@google.com>
parents: 43883
diff changeset
  1316
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1317
34357
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1318
class cowdict(cow, dict):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1319
    """copy-on-write dict
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1320
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1321
    Be sure to call d = d.preparewrite() before writing to d.
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1322
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1323
    >>> a = cowdict()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1324
    >>> a is a.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1325
    True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1326
    >>> b = a.copy()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1327
    >>> b is a
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1328
    True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1329
    >>> c = b.copy()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1330
    >>> c is a
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1331
    True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1332
    >>> a = a.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1333
    >>> b is a
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1334
    False
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1335
    >>> a is a.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1336
    True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1337
    >>> c = c.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1338
    >>> b is c
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1339
    False
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1340
    >>> b is b.preparewrite()
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1341
    True
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1342
    """
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1343
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1344
34357
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1345
class cowsortdict(cow, sortdict):
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1346
    """copy-on-write sortdict
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1347
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1348
    Be sure to call d = d.preparewrite() before writing to d.
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1349
    """
c41444a39de2 config: use copy-on-write to improve copy performance
Jun Wu <quark@fb.com>
parents: 34295
diff changeset
  1350
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1351
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
  1352
class transactional:  # pytype: disable=ignored-metaclass
33793
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1353
    """Base class for making a transactional type into a context manager."""
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1354
33793
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1355
    __metaclass__ = abc.ABCMeta
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1356
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1357
    @abc.abstractmethod
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1358
    def close(self):
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1359
        """Successfully closes the transaction."""
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1360
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1361
    @abc.abstractmethod
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1362
    def release(self):
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1363
        """Marks the end of the transaction.
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1364
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1365
        If the transaction has not been closed, it will be aborted.
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1366
        """
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1367
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1368
    def __enter__(self):
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1369
        return self
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1370
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1371
    def __exit__(self, exc_type, exc_val, exc_tb):
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1372
        try:
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1373
            if exc_type is None:
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1374
                self.close()
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1375
        finally:
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1376
            self.release()
bbbbd3c30bfc util: add base class for transactional context managers
Martin von Zweigbergk <martinvonz@google.com>
parents: 33737
diff changeset
  1377
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1378
33446
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
  1379
@contextlib.contextmanager
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
  1380
def acceptintervention(tr=None):
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
  1381
    """A context manager that closes the transaction on InterventionRequired
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
  1382
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
  1383
    If no transaction was provided, this simply runs the body and returns
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
  1384
    """
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
  1385
    if not tr:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
  1386
        yield
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
  1387
        return
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
  1388
    try:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
  1389
        yield
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
  1390
        tr.close()
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
  1391
    except error.InterventionRequired:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
  1392
        tr.close()
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
  1393
        raise
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
  1394
    finally:
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
  1395
        tr.release()
fad6852cf879 histedit: extract InterventionRequired transaction handling to utils
Martin von Zweigbergk <martinvonz@google.com>
parents: 33439
diff changeset
  1396
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1397
33619
609606d21765 rebase: use one dirstateguard for when using rebase.singletransaction
Durham Goode <durham@fb.com>
parents: 33549
diff changeset
  1398
@contextlib.contextmanager
45146
a0791bfd9cfa util: enhance `nullcontextmanager` to be able to return __enter__ result
Manuel Jacob <me@manueljacob.de>
parents: 45061
diff changeset
  1399
def nullcontextmanager(enter_result=None):
a0791bfd9cfa util: enhance `nullcontextmanager` to be able to return __enter__ result
Manuel Jacob <me@manueljacob.de>
parents: 45061
diff changeset
  1400
    yield enter_result
33619
609606d21765 rebase: use one dirstateguard for when using rebase.singletransaction
Durham Goode <durham@fb.com>
parents: 33549
diff changeset
  1401
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1402
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
  1403
class _lrucachenode:
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1404
    """A node in a doubly linked list.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1405
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1406
    Holds a reference to nodes on either side as well as a key-value
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1407
    pair for the dictionary entry.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1408
    """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1409
43506
9f70512ae2cf cleanup: remove pointless r-prefixes on single-quoted strings
Augie Fackler <augie@google.com>
parents: 43494
diff changeset
  1410
    __slots__ = ('next', 'prev', 'key', 'value', 'cost')
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1411
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1412
    def __init__(self):
46901
51841b23670b typing: make minor adjustments to mercurial/util.py to pass pytype checking
Matt Harbison <matt_harbison@yahoo.com>
parents: 46900
diff changeset
  1413
        self.next = self
51841b23670b typing: make minor adjustments to mercurial/util.py to pass pytype checking
Matt Harbison <matt_harbison@yahoo.com>
parents: 46900
diff changeset
  1414
        self.prev = self
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1415
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1416
        self.key = _notset
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1417
        self.value = None
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1418
        self.cost = 0
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1419
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1420
    def markempty(self):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1421
        """Mark the node as emptied."""
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1422
        self.key = _notset
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1423
        self.value = None
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1424
        self.cost = 0
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1425
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1426
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
  1427
class lrucachedict:
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1428
    """Dict that caches most recent accesses and sets.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1429
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1430
    The dict consists of an actual backing dict - indexed by original
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1431
    key - and a doubly linked circular list defining the order of entries in
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1432
    the cache.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1433
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1434
    The head node is the newest entry in the cache. If the cache is full,
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1435
    we recycle head.prev and make it the new head. Cache accesses result in
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1436
    the node being moved to before the existing head and being marked as the
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1437
    new head node.
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1438
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1439
    Items in the cache can be inserted with an optional "cost" value. This is
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1440
    simply an integer that is specified by the caller. The cache can be queried
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1441
    for the total cost of all items presently in the cache.
39568
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1442
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1443
    The cache can also define a maximum cost. If a cache insertion would
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1444
    cause the total cost of the cache to go beyond the maximum cost limit,
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1445
    nodes will be evicted to make room for the new code. This can be used
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1446
    to e.g. set a max memory limit and associate an estimated bytes size
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1447
    cost to each item in the cache. By default, no maximum cost is enforced.
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1448
    """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1449
39568
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1450
    def __init__(self, max, maxcost=0):
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
  1451
        self._cache = {}
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1452
46901
51841b23670b typing: make minor adjustments to mercurial/util.py to pass pytype checking
Matt Harbison <matt_harbison@yahoo.com>
parents: 46900
diff changeset
  1453
        self._head = _lrucachenode()
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1454
        self._size = 1
39564
5d75a3c16193 util: make capacity a public attribute on lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39563
diff changeset
  1455
        self.capacity = max
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1456
        self.totalcost = 0
39568
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1457
        self.maxcost = maxcost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1458
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1459
    def __len__(self):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1460
        return len(self._cache)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1461
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1462
    def __contains__(self, k):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1463
        return k in self._cache
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
  1464
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1465
    def __iter__(self):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1466
        # We don't have to iterate in cache order, but why not.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1467
        n = self._head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1468
        for i in range(len(self._cache)):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1469
            yield n.key
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1470
            n = n.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1471
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1472
    def __getitem__(self, k):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1473
        node = self._cache[k]
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1474
        self._movetohead(node)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1475
        return node.value
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1476
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1477
    def insert(self, k, v, cost=0):
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1478
        """Insert a new item in the cache with optional cost value."""
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1479
        node = self._cache.get(k)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1480
        # Replace existing value and mark as newest.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1481
        if node is not None:
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1482
            self.totalcost -= node.cost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1483
            node.value = v
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1484
            node.cost = cost
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1485
            self.totalcost += cost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1486
            self._movetohead(node)
39568
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1487
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1488
            if self.maxcost:
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1489
                self._enforcecostlimit()
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1490
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1491
            return
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1492
39564
5d75a3c16193 util: make capacity a public attribute on lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39563
diff changeset
  1493
        if self._size < self.capacity:
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1494
            node = self._addcapacity()
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1495
        else:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1496
            # Grab the last/oldest item.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1497
            node = self._head.prev
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
  1498
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1499
        # At capacity. Kill the old entry.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1500
        if node.key is not _notset:
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1501
            self.totalcost -= node.cost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1502
            del self._cache[node.key]
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1503
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1504
        node.key = k
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1505
        node.value = v
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1506
        node.cost = cost
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1507
        self.totalcost += cost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1508
        self._cache[k] = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1509
        # And mark it as newest entry. No need to adjust order since it
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1510
        # is already self._head.prev.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1511
        self._head = node
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
  1512
39568
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1513
        if self.maxcost:
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1514
            self._enforcecostlimit()
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1515
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1516
    def __setitem__(self, k, v):
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1517
        self.insert(k, v)
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1518
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1519
    def __delitem__(self, k):
40880
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
  1520
        self.pop(k)
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
  1521
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
  1522
    def pop(self, k, default=_notset):
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
  1523
        try:
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
  1524
            node = self._cache.pop(k)
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
  1525
        except KeyError:
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
  1526
            if default is _notset:
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
  1527
                raise
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
  1528
            return default
43881
b5655f337bd7 typing: add a couple of assertions to lrucachedict to help pytype
Matt Harbison <matt_harbison@yahoo.com>
parents: 43880
diff changeset
  1529
40880
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
  1530
        value = node.value
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1531
        self.totalcost -= node.cost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1532
        node.markempty()
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1533
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1534
        # Temporarily mark as newest item before re-adjusting head to make
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1535
        # this node the oldest item.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1536
        self._movetohead(node)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1537
        self._head = node.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1538
40880
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
  1539
        return value
7cda0cacbbf6 util: implement pop() on lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40879
diff changeset
  1540
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1541
    # Additional dict methods.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1542
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1543
    def get(self, k, default=None):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1544
        try:
39571
8f2c0d1b454c util: update lrucachedict order during get()
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39570
diff changeset
  1545
            return self.__getitem__(k)
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1546
        except KeyError:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1547
            return default
18603
2251b3184e6e util: add an LRU cache dict
Siddharth Agarwal <sid0@fb.com>
parents: 18537
diff changeset
  1548
40879
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
  1549
    def peek(self, k, default=_notset):
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
  1550
        """Get the specified item without moving it to the head
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
  1551
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
  1552
        Unlike get(), this doesn't mutate the internal state. But be aware
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
  1553
        that it doesn't mean peek() is thread safe.
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
  1554
        """
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
  1555
        try:
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
  1556
            node = self._cache[k]
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
  1557
            return node.value
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
  1558
        except KeyError:
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
  1559
            if default is _notset:
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
  1560
                raise
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
  1561
            return default
0c638ff69f5c util: add method to peek item in lrucachedict
Yuya Nishihara <yuya@tcha.org>
parents: 40689
diff changeset
  1562
19710
887ffa22fd0d lrucachedict: implement clear()
Siddharth Agarwal <sid0@fb.com>
parents: 19461
diff changeset
  1563
    def clear(self):
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1564
        n = self._head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1565
        while n.key is not _notset:
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1566
            self.totalcost -= n.cost
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1567
            n.markempty()
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1568
            n = n.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1569
19710
887ffa22fd0d lrucachedict: implement clear()
Siddharth Agarwal <sid0@fb.com>
parents: 19461
diff changeset
  1570
        self._cache.clear()
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1571
39568
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1572
    def copy(self, capacity=None, maxcost=0):
39565
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
  1573
        """Create a new cache as a copy of the current one.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
  1574
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
  1575
        By default, the new cache has the same capacity as the existing one.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
  1576
        But, the cache capacity can be changed as part of performing the
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
  1577
        copy.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
  1578
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
  1579
        Items in the copy have an insertion/access order matching this
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
  1580
        instance.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
  1581
        """
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
  1582
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
  1583
        capacity = capacity or self.capacity
39568
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1584
        maxcost = maxcost or self.maxcost
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1585
        result = lrucachedict(capacity, maxcost=maxcost)
39563
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
  1586
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
  1587
        # We copy entries by iterating in oldest-to-newest order so the copy
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
  1588
        # has the correct ordering.
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
  1589
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
  1590
        # Find the first non-empty entry.
27576
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
  1591
        n = self._head.prev
39563
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
  1592
        while n.key is _notset and n is not self._head:
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
  1593
            n = n.prev
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
  1594
39565
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
  1595
        # We could potentially skip the first N items when decreasing capacity.
2dcc68c7d25b util: ability to change capacity when copying lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39564
diff changeset
  1596
        # But let's keep it simple unless it is a performance problem.
27576
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
  1597
        for i in range(len(self._cache)):
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1598
            result.insert(n.key, n.value, cost=n.cost)
27576
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
  1599
            n = n.prev
39563
b31b01f93b11 util: properly copy lrucachedict instances
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39259
diff changeset
  1600
27576
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
  1601
        return result
6cd3044985c2 lrucachedict: add copy method
Eric Sumner <ericsumner@fb.com>
parents: 27391
diff changeset
  1602
39566
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1603
    def popoldest(self):
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1604
        """Remove the oldest item from the cache.
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1605
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1606
        Returns the (key, value) describing the removed cache entry.
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1607
        """
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1608
        if not self._cache:
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1609
            return
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1610
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1611
        # Walk the linked list backwards starting at tail node until we hit
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1612
        # a non-empty node.
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1613
        n = self._head.prev
46901
51841b23670b typing: make minor adjustments to mercurial/util.py to pass pytype checking
Matt Harbison <matt_harbison@yahoo.com>
parents: 46900
diff changeset
  1614
39566
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1615
        while n.key is _notset:
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1616
            n = n.prev
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1617
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1618
        key, value = n.key, n.value
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1619
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1620
        # And remove it from the cache and mark it as empty.
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1621
        del self._cache[n.key]
39567
ee087f0d7db5 util: allow lrucachedict to track cost of entries
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39566
diff changeset
  1622
        self.totalcost -= n.cost
39566
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1623
        n.markempty()
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1624
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1625
        return key, value
bd9d3a89f07b util: add a popoldest() method to lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39565
diff changeset
  1626
51298
c8a2fdf5ca37 pytype: add the couple annotations for pytype to understands the lrunode
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51294
diff changeset
  1627
    def _movetohead(self, node: _lrucachenode):
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1628
        """Mark a node as the newest, making it the new head.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1629
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1630
        When a node is accessed, it becomes the freshest entry in the LRU
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1631
        list, which is denoted by self._head.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1632
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1633
        Visually, let's make ``N`` the new head node (* denotes head):
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1634
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1635
            previous/oldest <-> head <-> next/next newest
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1636
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1637
            ----<->--- A* ---<->-----
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1638
            |                       |
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1639
            E <-> D <-> N <-> C <-> B
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1640
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1641
        To:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1642
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1643
            ----<->--- N* ---<->-----
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1644
            |                       |
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1645
            E <-> D <-> C <-> B <-> A
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1646
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1647
        This requires the following moves:
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1648
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1649
           C.next = D  (node.prev.next = node.next)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1650
           D.prev = C  (node.next.prev = node.prev)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1651
           E.next = N  (head.prev.next = node)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1652
           N.prev = E  (node.prev = head.prev)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1653
           N.next = A  (node.next = head)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1654
           A.prev = N  (head.prev = node)
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1655
        """
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1656
        head = self._head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1657
        # C.next = D
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1658
        node.prev.next = node.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1659
        # D.prev = C
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1660
        node.next.prev = node.prev
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1661
        # N.prev = E
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1662
        node.prev = head.prev
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1663
        # N.next = A
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1664
        # It is tempting to do just "head" here, however if node is
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1665
        # adjacent to head, this will do bad things.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1666
        node.next = head.prev.next
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1667
        # E.next = N
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1668
        node.next.prev = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1669
        # A.prev = N
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1670
        node.prev.next = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1671
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1672
        self._head = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1673
51298
c8a2fdf5ca37 pytype: add the couple annotations for pytype to understands the lrunode
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51294
diff changeset
  1674
    def _addcapacity(self) -> _lrucachenode:
27371
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1675
        """Add a node to the circular linked list.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1676
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1677
        The new node is inserted before the head node.
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1678
        """
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1679
        head = self._head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1680
        node = _lrucachenode()
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1681
        head.prev.next = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1682
        node.prev = head.prev
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1683
        node.next = head
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1684
        head.prev = node
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1685
        self._size += 1
45d996a566d7 util: reimplement lrucachedict
Gregory Szorc <gregory.szorc@gmail.com>
parents: 27363
diff changeset
  1686
        return node
19710
887ffa22fd0d lrucachedict: implement clear()
Siddharth Agarwal <sid0@fb.com>
parents: 19461
diff changeset
  1687
39568
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1688
    def _enforcecostlimit(self):
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1689
        # This should run after an insertion. It should only be called if total
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1690
        # cost limits are being enforced.
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1691
        # The most recently inserted node is never evicted.
39569
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
  1692
        if len(self) <= 1 or self.totalcost <= self.maxcost:
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
  1693
            return
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
  1694
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
  1695
        # This is logically equivalent to calling popoldest() until we
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
  1696
        # free up enough cost. We don't do that since popoldest() needs
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
  1697
        # to walk the linked list and doing this in a loop would be
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
  1698
        # quadratic. So we find the first non-empty node and then
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
  1699
        # walk nodes until we free up enough capacity.
39570
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
  1700
        #
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
  1701
        # If we only removed the minimum number of nodes to free enough
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
  1702
        # cost at insert time, chances are high that the next insert would
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
  1703
        # also require pruning. This would effectively constitute quadratic
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
  1704
        # behavior for insert-heavy workloads. To mitigate this, we set a
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
  1705
        # target cost that is a percentage of the max cost. This will tend
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
  1706
        # to free more nodes when the high water mark is reached, which
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
  1707
        # lowers the chances of needing to prune on the subsequent insert.
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
  1708
        targetcost = int(self.maxcost * 0.75)
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
  1709
39569
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
  1710
        n = self._head.prev
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
  1711
        while n.key is _notset:
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
  1712
            n = n.prev
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
  1713
39570
f296c0b366c8 util: lower water mark when removing nodes after cost limit reached
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39569
diff changeset
  1714
        while len(self) > 1 and self.totalcost > targetcost:
39569
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
  1715
            del self._cache[n.key]
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
  1716
            self.totalcost -= n.cost
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
  1717
            n.markempty()
cc23c09bc562 util: optimize cost auditing on insert
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39568
diff changeset
  1718
            n = n.prev
39568
842cd0bdda75 util: teach lrucachedict to enforce a max total cost
Gregory Szorc <gregory.szorc@gmail.com>
parents: 39567
diff changeset
  1719
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1720
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1721
def lrucachefunc(func):
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1722
    '''cache most recent results of function calls'''
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1723
    cache = {}
25113
0ca8410ea345 util: drop alias for collections.deque
Martin von Zweigbergk <martinvonz@google.com>
parents: 25112
diff changeset
  1724
    order = collections.deque()
28832
f5ff10f6fa6b util: use __code__ (available since py2.6)
timeless <timeless@mozdev.org>
parents: 28826
diff changeset
  1725
    if func.__code__.co_argcount == 1:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1726
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1727
        def f(arg):
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1728
            if arg not in cache:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1729
                if len(cache) > 20:
16803
107a3270a24a cleanup: use the deque type where appropriate
Bryan O'Sullivan <bryano@fb.com>
parents: 16769
diff changeset
  1730
                    del cache[order.popleft()]
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1731
                cache[arg] = func(arg)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1732
            else:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1733
                order.remove(arg)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1734
            order.append(arg)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1735
            return cache[arg]
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1736
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1737
    else:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1738
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1739
        def f(*args):
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1740
            if args not in cache:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1741
                if len(cache) > 20:
16803
107a3270a24a cleanup: use the deque type where appropriate
Bryan O'Sullivan <bryano@fb.com>
parents: 16769
diff changeset
  1742
                    del cache[order.popleft()]
9097
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1743
                cache[args] = func(*args)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1744
            else:
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1745
                order.remove(args)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1746
            order.append(args)
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1747
            return cache[args]
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1748
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1749
    return f
431462bd8478 fix memory usage of revlog caches by limiting cache size [issue1639]
Matt Mackall <mpm@selenic.com>
parents: 9089
diff changeset
  1750
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1751
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
  1752
class propertycache:
8207
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
  1753
    def __init__(self, func):
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
  1754
        self.func = func
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
  1755
        self.name = func.__name__
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1756
8207
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
  1757
    def __get__(self, obj, type=None):
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
  1758
        result = self.func(obj)
18013
98c867ac1330 clfilter: add a propertycache that must be unfiltered
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 17962
diff changeset
  1759
        self.cachevalue(obj, result)
8207
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
  1760
        return result
dd8d5be57d65 util: take propertycache from context.py
Matt Mackall <mpm@selenic.com>
parents: 8181
diff changeset
  1761
18013
98c867ac1330 clfilter: add a propertycache that must be unfiltered
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 17962
diff changeset
  1762
    def cachevalue(self, obj, value):
19951
d51c4d85ec23 spelling: random spell checker fixes
Mads Kiilerich <madski@unity3d.com>
parents: 19852
diff changeset
  1763
        # __dict__ assignment required to bypass __setattr__ (eg: repoview)
19845
a1237a4b437d repoview: make propertycache.setcache compatible with repoview
Pierre-Yves David <pierre-yves.david@ens-lyon.org>
parents: 19461
diff changeset
  1764
        obj.__dict__[self.name] = value
18013
98c867ac1330 clfilter: add a propertycache that must be unfiltered
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 17962
diff changeset
  1765
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1766
35014
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
  1767
def clearcachedproperty(obj, prop):
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
  1768
    '''clear a cached property value, if one has been set'''
40689
475921a3028c py3: cast attribute name to sysstr in clearcachedproperty()
Yuya Nishihara <yuya@tcha.org>
parents: 40251
diff changeset
  1769
    prop = pycompat.sysstr(prop)
35014
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
  1770
    if prop in obj.__dict__:
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
  1771
        del obj.__dict__[prop]
be6aa0cff8ea util: add util.clearcachedproperty
Mark Thomas <mbthomas@fb.com>
parents: 34647
diff changeset
  1772
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1773
7396
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1774
def increasingchunks(source, min=1024, max=65536):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  1775
    """return no less than min bytes per chunk while data remains,
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  1776
    doubling min after each chunk until it reaches max"""
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1777
7396
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1778
    def log2(x):
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1779
        if not x:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1780
            return 0
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1781
        i = 0
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1782
        while x:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1783
            x >>= 1
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1784
            i += 1
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1785
        return i - 1
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1786
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1787
    buf = []
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1788
    blen = 0
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1789
    for chunk in source:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1790
        buf.append(chunk)
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1791
        blen += len(chunk)
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1792
        if blen >= min:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1793
            if min < max:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1794
                min = min << 1
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1795
                nmin = 1 << log2(blen)
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1796
                if nmin > min:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1797
                    min = nmin
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1798
                if min > max:
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1799
                    min = max
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1800
            yield b''.join(buf)
7396
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1801
            blen = 0
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1802
            buf = []
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1803
    if buf:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1804
        yield b''.join(buf)
7396
526c40a74bd0 templater: return data in increasing chunk sizes
Brendan Cully <brendan@kublai.com>
parents: 7301
diff changeset
  1805
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1806
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
  1807
def always(fn):
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
  1808
    return True
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
  1809
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1810
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
  1811
def never(fn):
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
  1812
    return False
724
1c0c413cccdd Get add and locate to use new repo and dirstate walk code.
Bryan O'Sullivan <bos@serpentine.com>
parents: 705
diff changeset
  1813
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1814
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1815
def nogc(func):
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1816
    """disable garbage collector
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1817
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1818
    Python's garbage collector triggers a GC each time a certain number of
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1819
    container objects (the number being defined by gc.get_threshold()) are
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1820
    allocated even when marked not to be tracked by the collector. Tracking has
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1821
    no effect on when GCs are triggered, only on what objects the GC looks
23543
4dd8a6a1240d spelling: fixes from proofreading of spell checker issues
Mads Kiilerich <madski@unity3d.com>
parents: 23495
diff changeset
  1822
    into. As a workaround, disable GC while building complex (huge)
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1823
    containers.
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1824
33799
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33793
diff changeset
  1825
    This garbage collector issue have been fixed in 2.7. But it still affect
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33793
diff changeset
  1826
    CPython's performance.
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1827
    """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1828
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1829
    def wrapper(*args, **kwargs):
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1830
        gcenabled = gc.isenabled()
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1831
        gc.disable()
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1832
        try:
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1833
            return func(*args, **kwargs)
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1834
        finally:
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1835
            if gcenabled:
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1836
                gc.enable()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1837
23495
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1838
    return wrapper
b25f07cb5399 util: add a 'nogc' decorator to disable the garbage collection
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23370
diff changeset
  1839
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1840
33799
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33793
diff changeset
  1841
if pycompat.ispypy:
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33793
diff changeset
  1842
    # PyPy runs slower with gc disabled
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33793
diff changeset
  1843
    nogc = lambda x: x
05264fc9d8d6 util: make nogc effective for CPython
Jun Wu <quark@fb.com>
parents: 33793
diff changeset
  1844
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1845
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  1846
def pathto(root: bytes, n1: bytes, n2: bytes) -> bytes:
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  1847
    """return the relative path from one place to another.
4229
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
  1848
    root should use os.sep to separate directories
3669
48768b1ab23c fix util.pathto
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3629
diff changeset
  1849
    n1 should use os.sep to separate directories
48768b1ab23c fix util.pathto
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3629
diff changeset
  1850
    n2 should use "/" to separate directories
48768b1ab23c fix util.pathto
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3629
diff changeset
  1851
    returns an os.sep-separated path.
4229
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
  1852
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
  1853
    If n1 is a relative path, it's assumed it's
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
  1854
    relative to root.
24c22a3f2ef8 pass repo.root to util.pathto() in preparation for the next patch
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4129
diff changeset
  1855
    n2 should always be relative to root.
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  1856
    """
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
  1857
    if not n1:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
  1858
        return localpath(n2)
4230
c93562fb12cc Fix handling of paths when run outside the repo.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4229
diff changeset
  1859
    if os.path.isabs(n1):
c93562fb12cc Fix handling of paths when run outside the repo.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4229
diff changeset
  1860
        if os.path.splitdrive(root)[0] != os.path.splitdrive(n1)[0]:
c93562fb12cc Fix handling of paths when run outside the repo.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4229
diff changeset
  1861
            return os.path.join(root, localpath(n2))
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1862
        n2 = b'/'.join((pconvert(root), n2))
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1863
    a, b = splitpath(n1), n2.split(b'/')
1541
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX
twaldmann@thinkmo.de
parents: 1528
diff changeset
  1864
    a.reverse()
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX
twaldmann@thinkmo.de
parents: 1528
diff changeset
  1865
    b.reverse()
884
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
  1866
    while a and b and a[-1] == b[-1]:
1541
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX
twaldmann@thinkmo.de
parents: 1528
diff changeset
  1867
        a.pop()
bf4e7ef08741 fixed some stuff pychecker shows, marked unclear/wrong stuff with XXX
twaldmann@thinkmo.de
parents: 1528
diff changeset
  1868
        b.pop()
884
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
  1869
    b.reverse()
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1870
    return pycompat.ossep.join(([b'..'] * len(a)) + b) or b'.'
884
087771ebe2e6 Fix walk code for files that do not exist anywhere, and unhandled types.
Bryan O'Sullivan <bos@serpentine.com>
parents: 878
diff changeset
  1871
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1872
44436
09f3e003fc2a phabricator: avoid a stacktrace when command arguments are missing
Matt Harbison <matt_harbison@yahoo.com>
parents: 44060
diff changeset
  1873
def checksignature(func, depth=1):
7388
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
  1874
    '''wrap a function with code to check for calling errors'''
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1875
7388
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
  1876
    def check(*args, **kwargs):
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
  1877
        try:
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
  1878
            return func(*args, **kwargs)
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
  1879
        except TypeError:
44436
09f3e003fc2a phabricator: avoid a stacktrace when command arguments are missing
Matt Harbison <matt_harbison@yahoo.com>
parents: 44060
diff changeset
  1880
            if len(traceback.extract_tb(sys.exc_info()[2])) == depth:
7646
e62a456b8dc5 error: move SignatureError
Matt Mackall <mpm@selenic.com>
parents: 7644
diff changeset
  1881
                raise error.SignatureError
7388
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
  1882
            raise
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
  1883
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
  1884
    return check
5751631246de dispatch: generalize signature checking for extension command wrapping
Matt Mackall <mpm@selenic.com>
parents: 7301
diff changeset
  1885
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1886
31575
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31573
diff changeset
  1887
# a whilelist of known filesystems where hardlink works reliably
32291
bd872f64a8ba cleanup: use set literals
Martin von Zweigbergk <martinvonz@google.com>
parents: 32290
diff changeset
  1888
_hardlinkfswhitelist = {
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1889
    b'apfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1890
    b'btrfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1891
    b'ext2',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1892
    b'ext3',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1893
    b'ext4',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1894
    b'hfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1895
    b'jfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1896
    b'NTFS',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1897
    b'reiserfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1898
    b'tmpfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1899
    b'ufs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1900
    b'xfs',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1901
    b'zfs',
32291
bd872f64a8ba cleanup: use set literals
Martin von Zweigbergk <martinvonz@google.com>
parents: 32290
diff changeset
  1902
}
31575
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31573
diff changeset
  1903
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1904
47400
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47233
diff changeset
  1905
def copyfile(
47445
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47417
diff changeset
  1906
    src,
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47417
diff changeset
  1907
    dest,
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47417
diff changeset
  1908
    hardlink=False,
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47417
diff changeset
  1909
    copystat=False,
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47417
diff changeset
  1910
    checkambig=False,
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47417
diff changeset
  1911
    nb_bytes=None,
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47417
diff changeset
  1912
    no_hardlink_cb=None,
47446
09ff5d532a25 copyfiles: add a way to relax the file system checking for hardlink
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47445
diff changeset
  1913
    check_fs_hardlink=True,
47400
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47233
diff changeset
  1914
):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  1915
    """copy a file, preserving mode and optionally other stat info like
29367
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
  1916
    atime/mtime
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
  1917
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
  1918
    checkambig argument is used with filestat, and is useful only if
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
  1919
    destination file is guarded by any lock (e.g. repo.lock or
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
  1920
    repo.wlock).
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
  1921
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
  1922
    copystat and checkambig should be exclusive.
47400
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47233
diff changeset
  1923
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47233
diff changeset
  1924
    nb_bytes: if set only copy the first `nb_bytes` of the source file.
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  1925
    """
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
  1926
    assert not (copystat and checkambig)
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
  1927
    oldstat = None
18326
614f769e6aa7 util: copyfile: remove dest before copying
Mads Kiilerich <mads@kiilerich.com>
parents: 18026
diff changeset
  1928
    if os.path.lexists(dest):
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
  1929
        if checkambig:
32772
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
  1930
            oldstat = checkambig and filestat.frompath(dest)
18326
614f769e6aa7 util: copyfile: remove dest before copying
Mads Kiilerich <mads@kiilerich.com>
parents: 18026
diff changeset
  1931
        unlink(dest)
47446
09ff5d532a25 copyfiles: add a way to relax the file system checking for hardlink
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47445
diff changeset
  1932
    if hardlink and check_fs_hardlink:
31575
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31573
diff changeset
  1933
        # Hardlinks are problematic on CIFS (issue4546), do not allow hardlinks
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31573
diff changeset
  1934
        # unless we are confident that dest is on a whitelisted filesystem.
31678
1ed57a7dd904 statfs: make getfstype() raise OSError
Yuya Nishihara <yuya@tcha.org>
parents: 31662
diff changeset
  1935
        try:
1ed57a7dd904 statfs: make getfstype() raise OSError
Yuya Nishihara <yuya@tcha.org>
parents: 31662
diff changeset
  1936
            fstype = getfstype(os.path.dirname(dest))
1ed57a7dd904 statfs: make getfstype() raise OSError
Yuya Nishihara <yuya@tcha.org>
parents: 31662
diff changeset
  1937
        except OSError:
1ed57a7dd904 statfs: make getfstype() raise OSError
Yuya Nishihara <yuya@tcha.org>
parents: 31662
diff changeset
  1938
            fstype = None
31575
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31573
diff changeset
  1939
        if fstype not in _hardlinkfswhitelist:
47445
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47417
diff changeset
  1940
            if no_hardlink_cb is not None:
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47417
diff changeset
  1941
                no_hardlink_cb()
31575
e506e461c7a9 util: disable hardlink for copyfile if fstype is outside a whitelist
Jun Wu <quark@fb.com>
parents: 31573
diff changeset
  1942
            hardlink = False
31577
e7a02e9ad162 util: enable hardlink for copyfile
Jun Wu <quark@fb.com>
parents: 31575
diff changeset
  1943
    if hardlink:
23899
4e451d1359de copyfile: allow optional hardlinking
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23864
diff changeset
  1944
        try:
4e451d1359de copyfile: allow optional hardlinking
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23864
diff changeset
  1945
            oslink(src, dest)
47400
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47233
diff changeset
  1946
            if nb_bytes is not None:
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47233
diff changeset
  1947
                m = "the `nb_bytes` argument is incompatible with `hardlink`"
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47233
diff changeset
  1948
                raise error.ProgrammingError(m)
23899
4e451d1359de copyfile: allow optional hardlinking
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 23864
diff changeset
  1949
            return
47445
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47417
diff changeset
  1950
        except (IOError, OSError) as exc:
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47417
diff changeset
  1951
            if exc.errno != errno.EEXIST and no_hardlink_cb is not None:
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47417
diff changeset
  1952
                no_hardlink_cb()
d756fc11cfb9 copyfile: add a option callback for failed hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47417
diff changeset
  1953
            # fall back to normal copy
4271
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
  1954
    if os.path.islink(src):
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
  1955
        os.symlink(os.readlink(src), dest)
27369
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
  1956
        # copytime is ignored for symlinks, but in general copytime isn't needed
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
  1957
        # for them anyway
47400
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47233
diff changeset
  1958
        if nb_bytes is not None:
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47233
diff changeset
  1959
            m = "cannot use `nb_bytes` on a symlink"
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47233
diff changeset
  1960
            raise error.ProgrammingError(m)
4271
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
  1961
    else:
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
  1962
        try:
1eaa8d90c689 fix util.copyfile to deal with symlinks
Eric St-Jean <esj@wwd.ca>
parents: 4256
diff changeset
  1963
            shutil.copyfile(src, dest)
27369
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
  1964
            if copystat:
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
  1965
                # copystat also copies mode
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
  1966
                shutil.copystat(src, dest)
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
  1967
            else:
c48ecc0b5bc9 copyfile: add an optional parameter to copy other stat data
Siddharth Agarwal <sid0@fb.com>
parents: 26665
diff changeset
  1968
                shutil.copymode(src, dest)
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
  1969
                if oldstat and oldstat.stat:
32772
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
  1970
                    newstat = filestat.frompath(dest)
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
  1971
                    if newstat.isambig(oldstat):
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
  1972
                        # stat of copied file is ambiguous to original one
36781
ffa3026d4196 cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime
Augie Fackler <augie@google.com>
parents: 36724
diff changeset
  1973
                        advanced = (
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1974
                            oldstat.stat[stat.ST_MTIME] + 1
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1975
                        ) & 0x7FFFFFFF
29204
ce2d81aafbae util: make copyfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29201
diff changeset
  1976
                        os.utime(dest, (advanced, advanced))
47400
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47233
diff changeset
  1977
            # We could do something smarter using `copy_file_range` call or similar
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47233
diff changeset
  1978
            if nb_bytes is not None:
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47233
diff changeset
  1979
                with open(dest, mode='r+') as f:
9b841267253c util: add `nb_bytes` argument to `copyfile` to partially copy a file
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47233
diff changeset
  1980
                    f.truncate(nb_bytes)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
  1981
        except shutil.Error as inst:
43724
5be128f669d4 util: convert an exception to bytes when passing to Abort()
Matt Harbison <matt_harbison@yahoo.com>
parents: 43117
diff changeset
  1982
            raise error.Abort(stringutil.forcebytestr(inst))
3629
4cfb72bcb978 util: add copyfile function
Matt Mackall <mpm@selenic.com>
parents: 3568
diff changeset
  1983
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1984
38380
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38314
diff changeset
  1985
def copyfiles(src, dst, hardlink=None, progress=None):
24439
2ddfac2f163e util: add progress callback support to copyfiles
Augie Fackler <augie@google.com>
parents: 24236
diff changeset
  1986
    """Copy a directory tree using hardlinks if possible."""
2ddfac2f163e util: add progress callback support to copyfiles
Augie Fackler <augie@google.com>
parents: 24236
diff changeset
  1987
    num = 0
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
  1988
38380
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38314
diff changeset
  1989
    def settopic():
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38314
diff changeset
  1990
        if progress:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  1991
            progress.topic = _(b'linking') if hardlink else _(b'copying')
698
df78d8ccac4c Use python function instead of external 'cp' command when cloning repos.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 667
diff changeset
  1992
1207
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
  1993
    if os.path.isdir(src):
31719
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories
Jun Wu <quark@fb.com>
parents: 31718
diff changeset
  1994
        if hardlink is None:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1995
            hardlink = (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1996
                os.stat(src).st_dev == os.stat(os.path.dirname(dst)).st_dev
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  1997
            )
38380
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38314
diff changeset
  1998
        settopic()
1207
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
  1999
        os.mkdir(dst)
32208
d74b0cff94a9 osutil: proxy through util (and platform) modules (API)
Yuya Nishihara <yuya@tcha.org>
parents: 32201
diff changeset
  2000
        for name, kind in listdir(src):
1207
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
  2001
            srcname = os.path.join(src, name)
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
  2002
            dstname = os.path.join(dst, name)
38380
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38314
diff changeset
  2003
            hardlink, n = copyfiles(srcname, dstname, hardlink, progress)
11251
c61442f6d106 clone: print number of linked/copied files on --debug
Adrian Buehlmann <adrian@cadifra.com>
parents: 11232
diff changeset
  2004
            num += n
1207
a7b8812973d9 Rewrite copytree as copyfiles
mpm@selenic.com
parents: 1200
diff changeset
  2005
    else:
31719
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories
Jun Wu <quark@fb.com>
parents: 31718
diff changeset
  2006
        if hardlink is None:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2007
            hardlink = (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2008
                os.stat(os.path.dirname(src)).st_dev
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2009
                == os.stat(os.path.dirname(dst)).st_dev
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2010
            )
38380
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38314
diff changeset
  2011
        settopic()
31719
456efd1b51fd hardlink: duplicate hardlink detection for copying files and directories
Jun Wu <quark@fb.com>
parents: 31718
diff changeset
  2012
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
  2013
        if hardlink:
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
  2014
            try:
14235
b9e1b041744f rename util.os_link to oslink
Adrian Buehlmann <adrian@cadifra.com>
parents: 14234
diff changeset
  2015
                oslink(src, dst)
47417
9ea525216edb copyfiles: deal with existing file when hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47400
diff changeset
  2016
            except (IOError, OSError) as exc:
9ea525216edb copyfiles: deal with existing file when hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47400
diff changeset
  2017
                if exc.errno != errno.EEXIST:
9ea525216edb copyfiles: deal with existing file when hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47400
diff changeset
  2018
                    hardlink = False
9ea525216edb copyfiles: deal with existing file when hardlinking
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47400
diff changeset
  2019
                # XXX maybe try to relink if the file exist ?
1591
5a3229cf1492 do not copy atime and mtime in util.copyfiles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 1585
diff changeset
  2020
                shutil.copy(src, dst)
1241
3b4f05ff3130 Add support for cloning with hardlinks on windows.
Stephen Darnell
parents: 1207
diff changeset
  2021
        else:
1591
5a3229cf1492 do not copy atime and mtime in util.copyfiles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 1585
diff changeset
  2022
            shutil.copy(src, dst)
11251
c61442f6d106 clone: print number of linked/copied files on --debug
Adrian Buehlmann <adrian@cadifra.com>
parents: 11232
diff changeset
  2023
        num += 1
38380
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38314
diff changeset
  2024
        if progress:
63e6f5ae84bc copystore: use progress helper
Martin von Zweigbergk <martinvonz@google.com>
parents: 38314
diff changeset
  2025
            progress.increment()
698
df78d8ccac4c Use python function instead of external 'cp' command when cloning repos.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 667
diff changeset
  2026
11251
c61442f6d106 clone: print number of linked/copied files on --debug
Adrian Buehlmann <adrian@cadifra.com>
parents: 11232
diff changeset
  2027
    return hardlink, num
11254
640d419725d0 util.copyfiles: don't try os_link() again if it failed before
Adrian Buehlmann <adrian@cadifra.com>
parents: 11010
diff changeset
  2028
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2029
34052
ca6a3852daf0 util: use set for reserved Windows filenames
Gregory Szorc <gregory.szorc@gmail.com>
parents: 34002
diff changeset
  2030
_winreservednames = {
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2031
    b'con',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2032
    b'prn',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2033
    b'aux',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2034
    b'nul',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2035
    b'com1',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2036
    b'com2',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2037
    b'com3',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2038
    b'com4',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2039
    b'com5',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2040
    b'com6',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2041
    b'com7',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2042
    b'com8',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2043
    b'com9',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2044
    b'lpt1',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2045
    b'lpt2',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2046
    b'lpt3',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2047
    b'lpt4',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2048
    b'lpt5',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2049
    b'lpt6',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2050
    b'lpt7',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2051
    b'lpt8',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2052
    b'lpt9',
34052
ca6a3852daf0 util: use set for reserved Windows filenames
Gregory Szorc <gregory.szorc@gmail.com>
parents: 34002
diff changeset
  2053
}
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2054
_winreservedchars = b':*?"<>|'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2055
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2056
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2057
def checkwinfilename(path: bytes) -> Optional[bytes]:
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  2058
    r"""Check that the base-relative path is a valid filename on Windows.
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
  2059
    Returns None if the path is ok, or a UI string describing the problem.
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
  2060
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  2061
    >>> checkwinfilename(b"just/a/normal/path")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  2062
    >>> checkwinfilename(b"foo/bar/con.xml")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
  2063
    "filename contains 'con', which is reserved on Windows"
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  2064
    >>> checkwinfilename(b"foo/con.xml/bar")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
  2065
    "filename contains 'con', which is reserved on Windows"
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  2066
    >>> checkwinfilename(b"foo/bar/xml.con")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  2067
    >>> checkwinfilename(b"foo/bar/AUX/bla.txt")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
  2068
    "filename contains 'AUX', which is reserved on Windows"
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  2069
    >>> checkwinfilename(b"foo/bar/bla:.txt")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
  2070
    "filename contains ':', which is reserved on Windows"
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  2071
    >>> checkwinfilename(b"foo/bar/b\07la.txt")
20000
0849d280663e util: warn when adding paths ending with \
Mads Kiilerich <madski@unity3d.com>
parents: 19951
diff changeset
  2072
    "filename contains '\\x07', which is invalid on Windows"
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  2073
    >>> checkwinfilename(b"foo/bar/bla ")
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
  2074
    "filename ends with ' ', which is not allowed on Windows"
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  2075
    >>> checkwinfilename(b"../bar")
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  2076
    >>> checkwinfilename(b"foo\\")
20000
0849d280663e util: warn when adding paths ending with \
Mads Kiilerich <madski@unity3d.com>
parents: 19951
diff changeset
  2077
    "filename ends with '\\', which is invalid on Windows"
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  2078
    >>> checkwinfilename(b"foo\\/bar")
20000
0849d280663e util: warn when adding paths ending with \
Mads Kiilerich <madski@unity3d.com>
parents: 19951
diff changeset
  2079
    "directory name ends with '\\', which is invalid on Windows"
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  2080
    """
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2081
    if path.endswith(b'\\'):
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2082
        return _(b"filename ends with '\\', which is invalid on Windows")
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2083
    if b'\\/' in path:
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2084
        return _(b"directory name ends with '\\', which is invalid on Windows")
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2085
    for n in path.replace(b'\\', b'/').split(b'/'):
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
  2086
        if not n:
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
  2087
            continue
32572
377c74ef008d win32mbcs: avoid unintentional failure at colorization
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 31952
diff changeset
  2088
        for c in _filenamebytestr(n):
14262
23cd7eeff678 util: rename _windows_reserved_filenames and _windows_reserved_chars
Adrian Buehlmann <adrian@cadifra.com>
parents: 14250
diff changeset
  2089
            if c in _winreservedchars:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2090
                return (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2091
                    _(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2092
                        b"filename contains '%s', which is reserved "
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2093
                        b"on Windows"
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2094
                    )
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2095
                    % c
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2096
                )
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
  2097
            if ord(c) <= 31:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2098
                return _(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43106
diff changeset
  2099
                    b"filename contains '%s', which is invalid on Windows"
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2100
                ) % stringutil.escapestr(c)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2101
        base = n.split(b'.')[0]
14262
23cd7eeff678 util: rename _windows_reserved_filenames and _windows_reserved_chars
Adrian Buehlmann <adrian@cadifra.com>
parents: 14250
diff changeset
  2102
        if base and base.lower() in _winreservednames:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2103
            return (
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43106
diff changeset
  2104
                _(b"filename contains '%s', which is reserved on Windows")
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2105
                % base
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2106
            )
34361
7508a7dc95c1 py3: replace bytes[n] with slicing in checkwinfilename()
Yuya Nishihara <yuya@tcha.org>
parents: 34360
diff changeset
  2107
        t = n[-1:]
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2108
        if t in b'. ' and n not in b'..':
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2109
            return (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2110
                _(
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2111
                    b"filename ends with '%s', which is not allowed "
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2112
                    b"on Windows"
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2113
                )
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2114
                % t
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2115
            )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2116
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
  2117
44018
f9d29e1d3354 util: avoid referencing `time.clock()` on Windows when missing (issue6238)
Matt Harbison <matt_harbison@yahoo.com>
parents: 43920
diff changeset
  2118
timer = getattr(time, "perf_counter", None)
f9d29e1d3354 util: avoid referencing `time.clock()` on Windows when missing (issue6238)
Matt Harbison <matt_harbison@yahoo.com>
parents: 43920
diff changeset
  2119
34645
75979c8d4572 codemod: use pycompat.iswindows
Jun Wu <quark@fb.com>
parents: 34554
diff changeset
  2120
if pycompat.iswindows:
13916
98ee3dd5bab4 path_auditor: check filenames for basic platform validity (issue2755)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13910
diff changeset
  2121
    checkosfilename = checkwinfilename
44018
f9d29e1d3354 util: avoid referencing `time.clock()` on Windows when missing (issue6238)
Matt Harbison <matt_harbison@yahoo.com>
parents: 43920
diff changeset
  2122
    if not timer:
51294
7d3b92e8df13 pytype: ignore attribute error for time.clock
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51287
diff changeset
  2123
        timer = time.clock  # pytype: disable=module-attr
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
  2124
else:
43883
09bcbeacedc7 typing: suppress a warning that mercurial.windows.checkosfilename is missing
Matt Harbison <matt_harbison@yahoo.com>
parents: 43882
diff changeset
  2125
    # mercurial.windows doesn't have platform.checkosfilename
09bcbeacedc7 typing: suppress a warning that mercurial.windows.checkosfilename is missing
Matt Harbison <matt_harbison@yahoo.com>
parents: 43882
diff changeset
  2126
    checkosfilename = platform.checkosfilename  # pytype: disable=module-attr
44018
f9d29e1d3354 util: avoid referencing `time.clock()` on Windows when missing (issue6238)
Matt Harbison <matt_harbison@yahoo.com>
parents: 43920
diff changeset
  2127
    if not timer:
f9d29e1d3354 util: avoid referencing `time.clock()` on Windows when missing (issue6238)
Matt Harbison <matt_harbison@yahoo.com>
parents: 43920
diff changeset
  2128
        timer = time.time
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
  2129
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2130
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
  2131
def makelock(info, pathname):
36701
d77c3b023393 lock: block signal interrupt while making a lock file
Yuya Nishihara <yuya@tcha.org>
parents: 36699
diff changeset
  2132
    """Create a lock file atomically if possible
d77c3b023393 lock: block signal interrupt while making a lock file
Yuya Nishihara <yuya@tcha.org>
parents: 36699
diff changeset
  2133
d77c3b023393 lock: block signal interrupt while making a lock file
Yuya Nishihara <yuya@tcha.org>
parents: 36699
diff changeset
  2134
    This may leave a stale lock file if symlink isn't supported and signal
d77c3b023393 lock: block signal interrupt while making a lock file
Yuya Nishihara <yuya@tcha.org>
parents: 36699
diff changeset
  2135
    interrupt is enabled.
d77c3b023393 lock: block signal interrupt while making a lock file
Yuya Nishihara <yuya@tcha.org>
parents: 36699
diff changeset
  2136
    """
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
  2137
    try:
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
  2138
        return os.symlink(info, pathname)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
  2139
    except OSError as why:
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
  2140
        if why.errno == errno.EEXIST:
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
  2141
            raise
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2142
    except AttributeError:  # no symlink in os
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
  2143
        pass
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
  2144
36783
1fbbb8e83392 py3: read/write plain lock file in binary mode
Yuya Nishihara <yuya@tcha.org>
parents: 36782
diff changeset
  2145
    flags = os.O_CREAT | os.O_WRONLY | os.O_EXCL | getattr(os, 'O_BINARY', 0)
1fbbb8e83392 py3: read/write plain lock file in binary mode
Yuya Nishihara <yuya@tcha.org>
parents: 36782
diff changeset
  2146
    ld = os.open(pathname, flags)
704
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
  2147
    os.write(ld, info)
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
  2148
    os.close(ld)
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
  2149
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2150
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2151
def readlock(pathname: bytes) -> bytes:
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
  2152
    try:
39904
5fe0b880200e py3: convert os.readlink() path to native strings on Windows
Matt Harbison <matt_harbison@yahoo.com>
parents: 39777
diff changeset
  2153
        return readlink(pathname)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
  2154
    except OSError as why:
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
  2155
        if why.errno not in (errno.EINVAL, errno.ENOSYS):
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
  2156
            raise
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2157
    except AttributeError:  # no symlink in os
7890
e710f0f592b2 util: split out posix, windows, and win32 modules
Matt Mackall <mpm@selenic.com>
parents: 7879
diff changeset
  2158
        pass
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2159
    with posixfile(pathname, b'rb') as fp:
39905
4017968f0a1d util: use a context manager in readlock()
Matt Harbison <matt_harbison@yahoo.com>
parents: 39904
diff changeset
  2160
        return fp.read()
704
5ca319a641e1 Make makelock and readlock work on filesystems without symlink support.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 698
diff changeset
  2161
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2162
2176
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
  2163
def fstat(fp):
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
  2164
    '''stat file object that may not have fileno method.'''
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
  2165
    try:
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
  2166
        return os.fstat(fp.fileno())
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
  2167
    except AttributeError:
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
  2168
        return os.stat(fp.name)
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
  2169
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2170
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
  2171
# File system features
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
  2172
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2173
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2174
def fscasesensitive(path: bytes) -> bool:
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
  2175
    """
18911
451eb1c27c1b util: improve doc for checkcase
Mads Kiilerich <mads@kiilerich.com>
parents: 18868
diff changeset
  2176
    Return true if the given path is on a case-sensitive filesystem
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
  2177
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
  2178
    Requires a path (like /foo/.hg) ending with a foldable final
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
  2179
    directory component.
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
  2180
    """
24902
986a5c23b1c1 util.checkcase: don't abort on broken symlinks
Siddharth Agarwal <sid0@fb.com>
parents: 24692
diff changeset
  2181
    s1 = os.lstat(path)
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
  2182
    d, b = os.path.split(path)
15667
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
  2183
    b2 = b.upper()
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
  2184
    if b == b2:
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
  2185
        b2 = b.lower()
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
  2186
        if b == b2:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2187
            return True  # no evidence against case sensitivity
15667
eacfd851cb9e icasefs: consider as case sensitive if there is no counterevidence, for safety
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15609
diff changeset
  2188
    p2 = os.path.join(d, b2)
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
  2189
    try:
24902
986a5c23b1c1 util.checkcase: don't abort on broken symlinks
Siddharth Agarwal <sid0@fb.com>
parents: 24692
diff changeset
  2190
        s2 = os.lstat(p2)
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
  2191
        if s2 == s1:
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
  2192
            return False
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
  2193
        return True
14004
97ed99d1f419 eliminate various naked except clauses
Idan Kamara <idankk86@gmail.com>
parents: 13985
diff changeset
  2194
    except OSError:
3784
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
  2195
        return True
1427949b8f80 imported patch folding
Matt Mackall <mpm@selenic.com>
parents: 3770
diff changeset
  2196
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2197
46802
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46521
diff changeset
  2198
_re2_input = lambda x: x
16943
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
  2199
try:
43879
40bd667491a7 pytype: suppress the import-error in util.py when importing re2
Matt Harbison <matt_harbison@yahoo.com>
parents: 43754
diff changeset
  2200
    import re2  # pytype: disable=import-error
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2201
16943
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
  2202
    _re2 = None
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
  2203
except ImportError:
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
  2204
    _re2 = False
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
  2205
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2206
50689
a45460e235a2 re2: fix reporting of availability in `hg debuginstall`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50688
diff changeset
  2207
def has_re2():
a45460e235a2 re2: fix reporting of availability in `hg debuginstall`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50688
diff changeset
  2208
    """return True is re2 is available, False otherwise"""
a45460e235a2 re2: fix reporting of availability in `hg debuginstall`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50688
diff changeset
  2209
    if _re2 is None:
a45460e235a2 re2: fix reporting of availability in `hg debuginstall`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50688
diff changeset
  2210
        _re._checkre2()
a45460e235a2 re2: fix reporting of availability in `hg debuginstall`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50688
diff changeset
  2211
    return _re2
a45460e235a2 re2: fix reporting of availability in `hg debuginstall`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50688
diff changeset
  2212
a45460e235a2 re2: fix reporting of availability in `hg debuginstall`
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50688
diff changeset
  2213
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
  2214
class _re:
50688
293e1763982e re: make _checkre2 a static method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50687
diff changeset
  2215
    @staticmethod
293e1763982e re: make _checkre2 a static method
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50687
diff changeset
  2216
    def _checkre2():
21913
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
  2217
        global _re2
46802
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46521
diff changeset
  2218
        global _re2_input
50687
82cf392c99f6 re2: exit `_checkre2` early if calling it is useless
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49861
diff changeset
  2219
        if _re2 is not None:
82cf392c99f6 re2: exit `_checkre2` early if calling it is useless
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49861
diff changeset
  2220
            # we already have the answer
82cf392c99f6 re2: exit `_checkre2` early if calling it is useless
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 49861
diff changeset
  2221
            return
46926
9c3e84569071 re2: adjust local variable assignment scope
Matt Harbison <matt_harbison@yahoo.com>
parents: 46907
diff changeset
  2222
9c3e84569071 re2: adjust local variable assignment scope
Matt Harbison <matt_harbison@yahoo.com>
parents: 46907
diff changeset
  2223
        check_pattern = br'\[([^\[]+)\]'
9c3e84569071 re2: adjust local variable assignment scope
Matt Harbison <matt_harbison@yahoo.com>
parents: 46907
diff changeset
  2224
        check_input = b'[ui]'
21913
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
  2225
        try:
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
  2226
            # check if match works, see issue3964
46802
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46521
diff changeset
  2227
            _re2 = bool(re2.match(check_pattern, check_input))
21913
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
  2228
        except ImportError:
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
  2229
            _re2 = False
46802
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46521
diff changeset
  2230
        except TypeError:
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46521
diff changeset
  2231
            # the `pyre-2` project provides a re2 module that accept bytes
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46521
diff changeset
  2232
            # the `fb-re2` project provides a re2 module that acccept sysstr
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46521
diff changeset
  2233
            check_pattern = pycompat.sysstr(check_pattern)
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46521
diff changeset
  2234
            check_input = pycompat.sysstr(check_input)
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46521
diff changeset
  2235
            _re2 = bool(re2.match(check_pattern, check_input))
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46521
diff changeset
  2236
            _re2_input = pycompat.sysstr
21913
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
  2237
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
  2238
    def compile(self, pat, flags=0):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  2239
        """Compile a regular expression, using re2 if possible
16943
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
  2240
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
  2241
        For best performance, use only re2-compatible regexp features. The
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
  2242
        only flags from the re module that are re2-compatible are
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  2243
        IGNORECASE and MULTILINE."""
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
  2244
        if _re2 is None:
21913
50aad4609224 util.re: move check for re2 into a separate method
Siddharth Agarwal <sid0@fb.com>
parents: 21912
diff changeset
  2245
            self._checkre2()
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
  2246
        if _re2 and (flags & ~(remod.IGNORECASE | remod.MULTILINE)) == 0:
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
  2247
            if flags & remod.IGNORECASE:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2248
                pat = b'(?i)' + pat
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
  2249
            if flags & remod.MULTILINE:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2250
                pat = b'(?m)' + pat
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
  2251
            try:
46802
112826b59476 re2: feed unicode string to re2 module when necessary
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 46521
diff changeset
  2252
                return re2.compile(_re2_input(pat))
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
  2253
            except re2.error:
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
  2254
                pass
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
  2255
        return remod.compile(pat, flags)
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
  2256
21914
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
  2257
    @propertycache
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
  2258
    def escape(self):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  2259
        """Return the version of escape corresponding to self.compile.
21914
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
  2260
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
  2261
        This is imperfect because whether re2 or re is used for a particular
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
  2262
        function depends on the flags, etc, but it's the best we can do.
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  2263
        """
21914
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
  2264
        global _re2
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
  2265
        if _re2 is None:
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
  2266
            self._checkre2()
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
  2267
        if _re2:
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
  2268
            return re2.escape
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
  2269
        else:
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
  2270
            return remod.escape
10e99839a7a4 util.re: add an escape method
Siddharth Agarwal <sid0@fb.com>
parents: 21913
diff changeset
  2271
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2272
21908
cad9dadc9d26 util: move compilere to a class
Siddharth Agarwal <sid0@fb.com>
parents: 21907
diff changeset
  2273
re = _re()
16943
8d08a28aa63e matcher: use re2 bindings if available
Bryan O'Sullivan <bryano@fb.com>
parents: 16873
diff changeset
  2274
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
  2275
_fspathcache = {}
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2276
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2277
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2278
def fspath(name: bytes, root: bytes) -> bytes:
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  2279
    """Get name in the case stored in the filesystem
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
  2280
15710
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15709
diff changeset
  2281
    The name should be relative to root, and be normcase-ed for efficiency.
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15709
diff changeset
  2282
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15709
diff changeset
  2283
    Note that this function is unnecessary, and should not be
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
  2284
    called, for case-sensitive filesystems (simply because it's expensive).
15670
d6c19cfa03ce icasefs: avoid normcase()-ing in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15669
diff changeset
  2285
15710
f63e40047372 icasefs: avoid path-absoluteness/existance check in util.fspath() for efficiency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15709
diff changeset
  2286
    The root should be normcase-ed, too.
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  2287
    """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2288
23097
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
  2289
    def _makefspathcacheentry(dir):
44452
9d2b2df2c2ba cleanup: run pyupgrade on our source tree to clean up varying things
Augie Fackler <augie@google.com>
parents: 44436
diff changeset
  2290
        return {normcase(n): n for n in os.listdir(dir)}
15709
a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15674
diff changeset
  2291
30613
1112ff99d965 py3: replace os.sep with pycompat.ossep (part 1 of 4)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30536
diff changeset
  2292
    seps = pycompat.ossep
30625
bcf4a975f93d py3: replace os.altsep with pycompat.altsep
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30613
diff changeset
  2293
    if pycompat.osaltsep:
bcf4a975f93d py3: replace os.altsep with pycompat.altsep
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30613
diff changeset
  2294
        seps = seps + pycompat.osaltsep
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
  2295
    # Protect backslashes. This gets silly very quickly.
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2296
    seps.replace(b'\\', b'\\\\')
31496
670166e0fcaa util: use bytes re on bytes input in fspath
Augie Fackler <augie@google.com>
parents: 31495
diff changeset
  2297
    pattern = remod.compile(br'([^%s]+)|([%s]+)' % (seps, seps))
15669
390bcd01775a icasefs: use util.normcase() instead of lower() or os.path.normcase in fspath
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15667
diff changeset
  2298
    dir = os.path.normpath(root)
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
  2299
    result = []
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
  2300
    for part, sep in pattern.findall(name):
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
  2301
        if sep:
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
  2302
            result.append(sep)
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
  2303
            continue
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
  2304
15719
1dd60426b061 icasefs: follow standard cache look up pattern
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15718
diff changeset
  2305
        if dir not in _fspathcache:
23097
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
  2306
            _fspathcache[dir] = _makefspathcacheentry(dir)
15719
1dd60426b061 icasefs: follow standard cache look up pattern
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15718
diff changeset
  2307
        contents = _fspathcache[dir]
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
  2308
23097
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
  2309
        found = contents.get(part)
15709
a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15674
diff changeset
  2310
        if not found:
15720
3bcfea777efc icasefs: rewrite comment to explain situtation precisely
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15719
diff changeset
  2311
            # retry "once per directory" per "dirstate.walk" which
3bcfea777efc icasefs: rewrite comment to explain situtation precisely
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15719
diff changeset
  2312
            # may take place for each patches of "hg qpush", for example
23097
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
  2313
            _fspathcache[dir] = contents = _makefspathcacheentry(dir)
30124c40d11f util.fspath: use a dict rather than a linear scan for lookups
Siddharth Agarwal <sid0@fb.com>
parents: 23076
diff changeset
  2314
            found = contents.get(part)
15709
a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15674
diff changeset
  2315
a1f4bd47d18e icasefs: retry directory scan once for already invalidated cache
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15674
diff changeset
  2316
        result.append(found or part)
15669
390bcd01775a icasefs: use util.normcase() instead of lower() or os.path.normcase in fspath
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 15667
diff changeset
  2317
        dir = os.path.join(dir, part)
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
  2318
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2319
    return b''.join(result)
6676
33045179d079 Add a new function, fspath
Paul Moore <p.f.moore@gmail.com>
parents: 6595
diff changeset
  2320
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2321
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2322
def checknlink(testfile: bytes) -> bool:
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
  2323
    '''check whether hardlink count reporting works properly'''
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
  2324
13204
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
  2325
    # testfile may be open, so we need a separate file for checking to
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
  2326
    # work around issue2543 (or testfile may get lost on Samba shares)
34084
6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp'
Jun Wu <quark@fb.com>
parents: 34079
diff changeset
  2327
    f1, f2, fp = None, None, None
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
  2328
    try:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2329
        fd, f1 = pycompat.mkstemp(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2330
            prefix=b'.%s-' % os.path.basename(testfile),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2331
            suffix=b'1~',
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2332
            dir=os.path.dirname(testfile),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2333
        )
34079
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34072
diff changeset
  2334
        os.close(fd)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2335
        f2 = b'%s2~' % f1[:-2]
34079
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34072
diff changeset
  2336
25088
754df8e932d3 util: use try/except/finally
Matt Mackall <mpm@selenic.com>
parents: 24902
diff changeset
  2337
        oslink(f1, f2)
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
  2338
        # nlinks() may behave differently for files on Windows shares if
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
  2339
        # the file is open.
34084
6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp'
Jun Wu <quark@fb.com>
parents: 34079
diff changeset
  2340
        fp = posixfile(f2)
13204
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
  2341
        return nlinks(f2) > 1
25088
754df8e932d3 util: use try/except/finally
Matt Mackall <mpm@selenic.com>
parents: 24902
diff changeset
  2342
    except OSError:
754df8e932d3 util: use try/except/finally
Matt Mackall <mpm@selenic.com>
parents: 24902
diff changeset
  2343
        return False
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
  2344
    finally:
34084
6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp'
Jun Wu <quark@fb.com>
parents: 34079
diff changeset
  2345
        if fp is not None:
6c5cdb02f2f9 checknlink: rename file object from 'fd' to 'fp'
Jun Wu <quark@fb.com>
parents: 34079
diff changeset
  2346
            fp.close()
13204
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
  2347
        for f in (f1, f2):
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
  2348
            try:
34079
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34072
diff changeset
  2349
                if f is not None:
1104718fb090 checknlink: use a random temp file name for checking
Jun Wu <quark@fb.com>
parents: 34072
diff changeset
  2350
                    os.unlink(f)
13204
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
  2351
            except OSError:
5b83ab614dab checknlink: use two testfiles (issue2543)
Adrian Buehlmann <adrian@cadifra.com>
parents: 13188
diff changeset
  2352
                pass
12938
bf826c0b9537 opener: check hardlink count reporting (issue1866)
Adrian Buehlmann <adrian@cadifra.com>
parents: 12927
diff changeset
  2353
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2354
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2355
def endswithsep(path: bytes) -> bool:
5843
83c354c4d529 Add endswithsep() and use it instead of using os.sep and os.altsep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5802
diff changeset
  2356
    '''Check path ends with os.sep or os.altsep.'''
46645
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46113
diff changeset
  2357
    return bool(  # help pytype
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2358
        path.endswith(pycompat.ossep)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2359
        or pycompat.osaltsep
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2360
        and path.endswith(pycompat.osaltsep)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2361
    )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2362
5843
83c354c4d529 Add endswithsep() and use it instead of using os.sep and os.altsep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5802
diff changeset
  2363
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2364
def splitpath(path: bytes) -> List[bytes]:
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  2365
    """Split path by os.sep.
5844
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
  2366
    Note that this function does not use os.altsep because this is
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
  2367
    an alternative of simple "xxx.split(os.sep)".
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
  2368
    It is recommended to use os.path.normpath() before using this
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  2369
    function if need."""
30613
1112ff99d965 py3: replace os.sep with pycompat.ossep (part 1 of 4)
Pulkit Goyal <7895pulkit@gmail.com>
parents: 30536
diff changeset
  2370
    return path.split(pycompat.ossep)
5844
07d8eb78dd68 Add util.splitpath() and use it instead of using os.sep directly.
Shun-ichi GOTO <shunichi.goto@gmail.com>
parents: 5843
diff changeset
  2371
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2372
41289
593f6359681d update: fix edge-case with update.atomic-file and read-only files
Boris Feld <boris.feld@octobus.net>
parents: 40880
diff changeset
  2373
def mktempcopy(name, emptyok=False, createmode=None, enforcewritable=False):
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2374
    """Create a temporary file with the same contents from name
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2375
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2376
    The permission bits are copied from the original file.
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2377
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2378
    If the temporary file is going to be truncated immediately, you
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2379
    can use emptyok=True as an optimization.
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2380
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2381
    Returns the name of the temporary file.
2176
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
  2382
    """
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2383
    d, fn = os.path.split(name)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2384
    fd, temp = pycompat.mkstemp(prefix=b'.%s-' % fn, suffix=b'~', dir=d)
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2385
    os.close(fd)
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2386
    # Temporary files are created with mode 0600, which is usually not
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2387
    # what we want.  If the original file already exists, just copy
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2388
    # its mode.  Otherwise, manually obey umask.
41289
593f6359681d update: fix edge-case with update.atomic-file and read-only files
Boris Feld <boris.feld@octobus.net>
parents: 40880
diff changeset
  2389
    copymode(name, temp, createmode, enforcewritable)
593f6359681d update: fix edge-case with update.atomic-file and read-only files
Boris Feld <boris.feld@octobus.net>
parents: 40880
diff changeset
  2390
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2391
    if emptyok:
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2392
        return temp
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2393
    try:
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2394
        try:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2395
            ifp = posixfile(name, b"rb")
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
  2396
        except IOError as inst:
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2397
            if inst.errno == errno.ENOENT:
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2398
                return temp
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2399
            if not getattr(inst, 'filename', None):
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2400
                inst.filename = name
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2401
            raise
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2402
        ofp = posixfile(temp, b"wb")
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2403
        for chunk in filechunkiter(ifp):
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2404
            ofp.write(chunk)
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2405
        ifp.close()
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2406
        ofp.close()
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2407
    except:  # re-raises
34435
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34362
diff changeset
  2408
        try:
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34362
diff changeset
  2409
            os.unlink(temp)
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34362
diff changeset
  2410
        except OSError:
5326e4ef1dab style: never put multiple statements on one line
Alex Gaynor <agaynor@mozilla.com>
parents: 34362
diff changeset
  2411
            pass
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2412
        raise
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2413
    return temp
2176
9b42304d9896 fix file handling bugs on windows.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2117
diff changeset
  2414
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2415
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
  2416
class filestat:
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2417
    """help to exactly detect change of a file
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2418
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2419
    'stat' attribute is result of 'os.stat()' if specified 'path'
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2420
    exists. Otherwise, it is None. This can avoid preparative
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2421
    'exists()' examination on client side of this class.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2422
    """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2423
32772
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
  2424
    def __init__(self, stat):
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
  2425
        self.stat = stat
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
  2426
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
  2427
    @classmethod
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
  2428
    def frompath(cls, path):
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2429
        try:
32772
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
  2430
            stat = os.stat(path)
49306
2e726c934fcd py3: catch FileNotFoundError instead of checking errno == ENOENT
Manuel Jacob <me@manueljacob.de>
parents: 49275
diff changeset
  2431
        except FileNotFoundError:
32772
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
  2432
            stat = None
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
  2433
        return cls(stat)
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2434
32816
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32772
diff changeset
  2435
    @classmethod
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32772
diff changeset
  2436
    def fromfp(cls, fp):
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32772
diff changeset
  2437
        stat = os.fstat(fp.fileno())
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32772
diff changeset
  2438
        return cls(stat)
1b25c648d5b7 fsmonitor: don't write out state if identity has changed (issue5581)
Siddharth Agarwal <sid0@fb.com>
parents: 32772
diff changeset
  2439
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2440
    __hash__ = object.__hash__
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2441
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2442
    def __eq__(self, old):
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2443
        try:
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2444
            # if ambiguity between stat of new and old file is
30332
318a24b52eeb spelling: fixes of non-dictionary words
Mads Kiilerich <madski@unity3d.com>
parents: 30329
diff changeset
  2445
            # avoided, comparison of size, ctime and mtime is enough
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2446
            # to exactly detect change of a file regardless of platform
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2447
            return (
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2448
                self.stat.st_size == old.stat.st_size
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2449
                and self.stat[stat.ST_CTIME] == old.stat[stat.ST_CTIME]
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2450
                and self.stat[stat.ST_MTIME] == old.stat[stat.ST_MTIME]
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2451
            )
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2452
        except AttributeError:
32749
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32746
diff changeset
  2453
            pass
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32746
diff changeset
  2454
        try:
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32746
diff changeset
  2455
            return self.stat is None and old.stat is None
b5524fd9a4e3 util: make filestat.__eq__ return True if both of self and old have None stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32746
diff changeset
  2456
        except AttributeError:
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2457
            return False
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2458
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2459
    def isambig(self, old):
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2460
        """Examine whether new (= self) stat is ambiguous against old one
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2461
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2462
        "S[N]" below means stat of a file at N-th change:
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2463
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2464
        - S[n-1].ctime  < S[n].ctime: can detect change of a file
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2465
        - S[n-1].ctime == S[n].ctime
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2466
          - S[n-1].ctime  < S[n].mtime: means natural advancing (*1)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2467
          - S[n-1].ctime == S[n].mtime: is ambiguous (*2)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2468
          - S[n-1].ctime  > S[n].mtime: never occurs naturally (don't care)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2469
        - S[n-1].ctime  > S[n].ctime: never occurs naturally (don't care)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2470
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2471
        Case (*2) above means that a file was changed twice or more at
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2472
        same time in sec (= S[n-1].ctime), and comparison of timestamp
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2473
        is ambiguous.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2474
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2475
        Base idea to avoid such ambiguity is "advance mtime 1 sec, if
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2476
        timestamp is ambiguous".
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2477
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2478
        But advancing mtime only in case (*2) doesn't work as
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2479
        expected, because naturally advanced S[n].mtime in case (*1)
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2480
        might be equal to manually advanced S[n-1 or earlier].mtime.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2481
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2482
        Therefore, all "S[n-1].ctime == S[n].ctime" cases should be
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2483
        treated as ambiguous regardless of mtime, to avoid overlooking
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2484
        by confliction between such mtime.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2485
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2486
        Advancing mtime "if isambig(oldstat)" ensures "S[n-1].mtime !=
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2487
        S[n].mtime", even if size of a file isn't changed.
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2488
        """
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2489
        try:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2490
            return self.stat[stat.ST_CTIME] == old.stat[stat.ST_CTIME]
29200
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2491
        except AttributeError:
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2492
            return False
ca4065028e00 util: add filestat class to detect ambiguity of file stat
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29017
diff changeset
  2493
30319
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
  2494
    def avoidambig(self, path, old):
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
  2495
        """Change file stat of specified path to avoid ambiguity
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
  2496
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
  2497
        'old' should be previous filestat of 'path'.
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
  2498
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
  2499
        This skips avoiding ambiguity, if a process doesn't have
32746
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32652
diff changeset
  2500
        appropriate privileges for 'path'. This returns False in this
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32652
diff changeset
  2501
        case.
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32652
diff changeset
  2502
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32652
diff changeset
  2503
        Otherwise, this returns True, as "ambiguity is avoided".
30319
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
  2504
        """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2505
        advanced = (old.stat[stat.ST_MTIME] + 1) & 0x7FFFFFFF
30319
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
  2506
        try:
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
  2507
            os.utime(path, (advanced, advanced))
49308
d2adebe35635 py3: catch PermissionError instead of checking errno == EPERM
Manuel Jacob <me@manueljacob.de>
parents: 49306
diff changeset
  2508
        except PermissionError:
d2adebe35635 py3: catch PermissionError instead of checking errno == EPERM
Manuel Jacob <me@manueljacob.de>
parents: 49306
diff changeset
  2509
            # utime() on the file created by another user causes EPERM,
d2adebe35635 py3: catch PermissionError instead of checking errno == EPERM
Manuel Jacob <me@manueljacob.de>
parents: 49306
diff changeset
  2510
            # if a process doesn't have appropriate privileges
d2adebe35635 py3: catch PermissionError instead of checking errno == EPERM
Manuel Jacob <me@manueljacob.de>
parents: 49306
diff changeset
  2511
            return False
32746
77f354ae1123 util: make filestat.avoidambig() return whether ambiguity is avoided or not
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 32652
diff changeset
  2512
        return True
30319
b496a464399c util: add utility function to skip avoiding file stat ambiguity if EPERM
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 30181
diff changeset
  2513
29298
82f6193ff2bc util: add __ne__ to filestat class for consistency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29261
diff changeset
  2514
    def __ne__(self, other):
82f6193ff2bc util: add __ne__ to filestat class for consistency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29261
diff changeset
  2515
        return not self == other
82f6193ff2bc util: add __ne__ to filestat class for consistency
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29261
diff changeset
  2516
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2517
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
  2518
class atomictempfile:
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  2519
    """writable file object that atomically updates a file
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2520
14008
da65edcac72a atomictempfile: rewrite docstring to clarify rename() vs. close().
Greg Ward <greg@gerg.ca>
parents: 14007
diff changeset
  2521
    All writes will go to a temporary copy of the original file. Call
15057
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
  2522
    close() when you are done writing, and atomictempfile will rename
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
  2523
    the temporary copy to the original name, making the changes
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
  2524
    visible. If the object is destroyed without being closed, all your
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
  2525
    writes are discarded.
29367
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
  2526
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
  2527
    checkambig argument of constructor is used with filestat, and is
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
  2528
    useful only if target file is guarded by any lock (e.g. repo.lock
4e6e280e238f doc: describe detail about checkambig optional argument
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29342
diff changeset
  2529
    or repo.wlock).
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  2530
    """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2531
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2532
    def __init__(self, name, mode=b'w+b', createmode=None, checkambig=False):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2533
        self.__name = name  # permanent name
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2534
        self._tempname = mktempcopy(
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2535
            name,
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2536
            emptyok=(b'w' in mode),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2537
            createmode=createmode,
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2538
            enforcewritable=(b'w' in mode),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2539
        )
41289
593f6359681d update: fix edge-case with update.atomic-file and read-only files
Boris Feld <boris.feld@octobus.net>
parents: 40880
diff changeset
  2540
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
  2541
        self._fp = posixfile(self._tempname, mode)
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
  2542
        self._checkambig = checkambig
8327
aa25be1c2889 atomictempfile: delegate to posixfile instead of inheriting from it
Bryan O'Sullivan <bos@serpentine.com>
parents: 8312
diff changeset
  2543
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
  2544
        # delegated methods
29393
50269a4dce61 atomictempfile: add read to the supported file operations
Martijn Pieters <mjpieters@fb.com>
parents: 29367
diff changeset
  2545
        self.read = self._fp.read
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
  2546
        self.write = self._fp.write
49572
c4f07a011714 util: implement `writelines()` on atomictempfile
Matt Harbison <matt_harbison@yahoo.com>
parents: 49308
diff changeset
  2547
        self.writelines = self._fp.writelines
17237
e73128535105 util: delegate seek and tell methods of atomictempfile
Bryan O'Sullivan <bryano@fb.com>
parents: 17203
diff changeset
  2548
        self.seek = self._fp.seek
e73128535105 util: delegate seek and tell methods of atomictempfile
Bryan O'Sullivan <bryano@fb.com>
parents: 17203
diff changeset
  2549
        self.tell = self._fp.tell
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
  2550
        self.fileno = self._fp.fileno
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2551
15057
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
  2552
    def close(self):
8785
7a9151bc5b37 atomictempfile: fix exception in __del__ if mktempcopy fails (self._fp is None)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8778
diff changeset
  2553
        if not self._fp.closed:
8327
aa25be1c2889 atomictempfile: delegate to posixfile instead of inheriting from it
Bryan O'Sullivan <bos@serpentine.com>
parents: 8312
diff changeset
  2554
            self._fp.close()
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
  2555
            filename = localpath(self.__name)
32772
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
  2556
            oldstat = self._checkambig and filestat.frompath(filename)
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
  2557
            if oldstat and oldstat.stat:
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
  2558
                rename(self._tempname, filename)
32772
7ad95626f6a7 filestat: move __init__ to frompath constructor
Siddharth Agarwal <sid0@fb.com>
parents: 32749
diff changeset
  2559
                newstat = filestat.frompath(filename)
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
  2560
                if newstat.isambig(oldstat):
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
  2561
                    # stat of changed file is ambiguous to original one
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2562
                    advanced = (oldstat.stat[stat.ST_MTIME] + 1) & 0x7FFFFFFF
29201
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
  2563
                    os.utime(filename, (advanced, advanced))
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
  2564
            else:
a109bf7e0dc2 util: make atomictempfile avoid ambiguity of file stat if needed
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 29200
diff changeset
  2565
                rename(self._tempname, filename)
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2566
15057
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
  2567
    def discard(self):
8785
7a9151bc5b37 atomictempfile: fix exception in __del__ if mktempcopy fails (self._fp is None)
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 8778
diff changeset
  2568
        if not self._fp.closed:
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2569
            try:
14007
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
  2570
                os.unlink(self._tempname)
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
  2571
            except OSError:
d764463b433e atomictempfile: avoid infinite recursion in __del__().
Greg Ward <greg@gerg.ca>
parents: 14004
diff changeset
  2572
                pass
8327
aa25be1c2889 atomictempfile: delegate to posixfile instead of inheriting from it
Bryan O'Sullivan <bos@serpentine.com>
parents: 8312
diff changeset
  2573
            self._fp.close()
4827
89defeae88f3 turn util.opener into a class
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4803
diff changeset
  2574
13098
f7d6750dcd01 util: make atomicfiles closable
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13072
diff changeset
  2575
    def __del__(self):
50928
d718eddf01d9 safehasattr: drop usage in favor of hasattr
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 50753
diff changeset
  2576
        if hasattr(self, '_fp'):  # constructor actually did something
15057
774da7121fc9 atomictempfile: make close() consistent with other file-like objects.
Greg Ward <greg@gerg.ca>
parents: 15050
diff changeset
  2577
            self.discard()
13098
f7d6750dcd01 util: make atomicfiles closable
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 13072
diff changeset
  2578
29394
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
  2579
    def __enter__(self):
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
  2580
        return self
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
  2581
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
  2582
    def __exit__(self, exctype, excvalue, traceback):
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
  2583
        if exctype is not None:
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
  2584
            self.discard()
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
  2585
        else:
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
  2586
            self.close()
6d96658a22b0 atomictempfile: add context manager support
Martijn Pieters <mjpieters@fb.com>
parents: 29393
diff changeset
  2587
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2588
49190
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 48995
diff changeset
  2589
def tryrmdir(f):
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 48995
diff changeset
  2590
    try:
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 48995
diff changeset
  2591
        removedirs(f)
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 48995
diff changeset
  2592
    except OSError as e:
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 48995
diff changeset
  2593
        if e.errno != errno.ENOENT and e.errno != errno.ENOTEMPTY:
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 48995
diff changeset
  2594
            raise
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 48995
diff changeset
  2595
4ff4e23de7df clone: use better names for temp files
Arseniy Alekseyev <aalekseyev@janestreet.com>
parents: 48995
diff changeset
  2596
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2597
def unlinkpath(
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2598
    f: bytes, ignoremissing: bool = False, rmdir: bool = True
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2599
) -> None:
31539
52361c4f4dac util: unify unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31533
diff changeset
  2600
    """unlink and remove the directory if it is empty"""
31541
bd9daafbf87c util: use tryunlink in unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31540
diff changeset
  2601
    if ignoremissing:
bd9daafbf87c util: use tryunlink in unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31540
diff changeset
  2602
        tryunlink(f)
bd9daafbf87c util: use tryunlink in unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31540
diff changeset
  2603
    else:
31539
52361c4f4dac util: unify unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31533
diff changeset
  2604
        unlink(f)
38493
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38380
diff changeset
  2605
    if rmdir:
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38380
diff changeset
  2606
        # try removing directories that might now be empty
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38380
diff changeset
  2607
        try:
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38380
diff changeset
  2608
            removedirs(os.path.dirname(f))
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38380
diff changeset
  2609
        except OSError:
da2a7d8354b2 unlinkpath: make empty directory removal optional (issue5901) (issue5826)
Kyle Lippincott <spectral@google.com>
parents: 38380
diff changeset
  2610
            pass
31539
52361c4f4dac util: unify unlinkpath
Ryan McElroy <rmcelroy@fb.com>
parents: 31533
diff changeset
  2611
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2612
51430
187c5769a629 vfs: have tryunlink tell what it did
Georges Racinet <georges.racinet@octobus.net>
parents: 51298
diff changeset
  2613
def tryunlink(f: bytes) -> bool:
187c5769a629 vfs: have tryunlink tell what it did
Georges Racinet <georges.racinet@octobus.net>
parents: 51298
diff changeset
  2614
    """Attempt to remove a file, ignoring FileNotFoundError.
187c5769a629 vfs: have tryunlink tell what it did
Georges Racinet <georges.racinet@octobus.net>
parents: 51298
diff changeset
  2615
187c5769a629 vfs: have tryunlink tell what it did
Georges Racinet <georges.racinet@octobus.net>
parents: 51298
diff changeset
  2616
    Returns False in case the file did not exit, True otherwise
187c5769a629 vfs: have tryunlink tell what it did
Georges Racinet <georges.racinet@octobus.net>
parents: 51298
diff changeset
  2617
    """
31540
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
  2618
    try:
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
  2619
        unlink(f)
51430
187c5769a629 vfs: have tryunlink tell what it did
Georges Racinet <georges.racinet@octobus.net>
parents: 51298
diff changeset
  2620
        return True
49306
2e726c934fcd py3: catch FileNotFoundError instead of checking errno == ENOENT
Manuel Jacob <me@manueljacob.de>
parents: 49275
diff changeset
  2621
    except FileNotFoundError:
51430
187c5769a629 vfs: have tryunlink tell what it did
Georges Racinet <georges.racinet@octobus.net>
parents: 51298
diff changeset
  2622
        return False
31540
6d5b77abf306 util: add tryunlink function
Ryan McElroy <rmcelroy@fb.com>
parents: 31539
diff changeset
  2623
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2624
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2625
def makedirs(
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2626
    name: bytes, mode: Optional[int] = None, notindexed: bool = False
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2627
) -> None:
29017
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
  2628
    """recursive directory creation with parent mode inheritance
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
  2629
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
  2630
    Newly created directories are marked as "not to be indexed by
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
  2631
    the content indexing service", if ``notindexed`` is specified
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
  2632
    for "write" mode access.
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
  2633
    """
6062
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 6007
diff changeset
  2634
    try:
18938
e22107cff6bf util: add notindexed optional parameter to makedirs function
Angel Ezquerra <angel.ezquerra@gmail.com>
parents: 18930
diff changeset
  2635
        makedir(name, notindexed)
25660
328739ea70c3 global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents: 25420
diff changeset
  2636
    except OSError as err:
6062
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 6007
diff changeset
  2637
        if err.errno == errno.EEXIST:
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 6007
diff changeset
  2638
            return
15058
81f33be0ea79 util: postpone and reorder parent calculation in makedirs
Adrian Buehlmann <adrian@cadifra.com>
parents: 15057
diff changeset
  2639
        if err.errno != errno.ENOENT or not name:
81f33be0ea79 util: postpone and reorder parent calculation in makedirs
Adrian Buehlmann <adrian@cadifra.com>
parents: 15057
diff changeset
  2640
            raise
47622
bb917eea1605 windows: introduce a `util.abspath` to replace os.path.abspath
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 47446
diff changeset
  2641
        parent = os.path.dirname(abspath(name))
15058
81f33be0ea79 util: postpone and reorder parent calculation in makedirs
Adrian Buehlmann <adrian@cadifra.com>
parents: 15057
diff changeset
  2642
        if parent == name:
6062
3c3b126e5619 Make files in .hg inherit the permissions from .hg/store
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 6007
diff changeset
  2643
            raise
18938
e22107cff6bf util: add notindexed optional parameter to makedirs function
Angel Ezquerra <angel.ezquerra@gmail.com>
parents: 18930
diff changeset
  2644
        makedirs(parent, mode, notindexed)
29017
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
  2645
        try:
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
  2646
            makedir(name, notindexed)
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
  2647
        except OSError as err:
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
  2648
            # Catch EEXIST to handle races
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
  2649
            if err.errno == errno.EEXIST:
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
  2650
                return
07be86828e79 util: fix race in makedirs()
Adam Simpkins <simpkins@fb.com>
parents: 28883
diff changeset
  2651
            raise
18678
423eee0b0b14 util: make ensuredirs safer against races
Bryan O'Sullivan <bryano@fb.com>
parents: 18668
diff changeset
  2652
    if mode is not None:
423eee0b0b14 util: make ensuredirs safer against races
Bryan O'Sullivan <bryano@fb.com>
parents: 18668
diff changeset
  2653
        os.chmod(name, mode)
18668
4034b8d551b1 scmutil: create directories in a race-safe way during update
Bryan O'Sullivan <bryano@fb.com>
parents: 18614
diff changeset
  2654
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2655
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2656
def readfile(path: bytes) -> bytes:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2657
    with open(path, b'rb') as fp:
14100
3e9e02a41dfb util: really drop size from readfile
Matt Mackall <mpm@selenic.com>
parents: 14099
diff changeset
  2658
        return fp.read()
14099
0824a0a3cefc util: add readfile() & writefile() helper functions
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14077
diff changeset
  2659
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2660
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2661
def writefile(path: bytes, text: bytes) -> None:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2662
    with open(path, b'wb') as fp:
14167
0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14147
diff changeset
  2663
        fp.write(text)
0e4753807c93 util & scmutil: adapt read/write helpers as request by mpm
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14147
diff changeset
  2664
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2665
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2666
def appendfile(path: bytes, text: bytes) -> None:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2667
    with open(path, b'ab') as fp:
14099
0824a0a3cefc util: add readfile() & writefile() helper functions
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14077
diff changeset
  2668
        fp.write(text)
0824a0a3cefc util: add readfile() & writefile() helper functions
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 14077
diff changeset
  2669
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2670
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
  2671
class chunkbuffer:
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
  2672
    """Allow arbitrary sized chunks of data to be efficiently read from an
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
  2673
    iterator over chunks of arbitrary size."""
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
  2674
5446
fa836e050c50 chunkbuffer: removed unused method and arg
Matt Mackall <mpm@selenic.com>
parents: 5420
diff changeset
  2675
    def __init__(self, in_iter):
32123
5f53c267e362 util: remove doc of long gone 'targetsize' argument
Martin von Zweigbergk <martinvonz@google.com>
parents: 31952
diff changeset
  2676
        """in_iter is the iterator that's iterating over the input chunks."""
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2677
11670
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
  2678
        def splitbig(chunks):
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
  2679
            for chunk in chunks:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2680
                if len(chunk) > 2 ** 20:
11670
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
  2681
                    pos = 0
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
  2682
                    while pos < len(chunk):
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
  2683
                        end = pos + 2 ** 18
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
  2684
                        yield chunk[pos:end]
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
  2685
                        pos = end
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
  2686
                else:
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
  2687
                    yield chunk
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2688
11670
1b3b843e1100 chunkbuffer: split big strings directly in chunkbuffer
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 11668
diff changeset
  2689
        self.iter = splitbig(in_iter)
25113
0ca8410ea345 util: drop alias for collections.deque
Martin von Zweigbergk <martinvonz@google.com>
parents: 25112
diff changeset
  2690
        self._queue = collections.deque()
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2691
        self._chunkoffset = 0
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
  2692
21018
c848bfd02366 util: support None size in chunkbuffer.read()
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20835
diff changeset
  2693
    def read(self, l=None):
1200
333de1d53846 Minor cleanups.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1199
diff changeset
  2694
        """Read L bytes of data from the iterator of chunks of data.
21018
c848bfd02366 util: support None size in chunkbuffer.read()
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20835
diff changeset
  2695
        Returns less than L bytes if the iterator runs dry.
c848bfd02366 util: support None size in chunkbuffer.read()
Pierre-Yves David <pierre-yves.david@fb.com>
parents: 20835
diff changeset
  2696
23139
e53f6b72a0e4 spelling: fixes from proofreading of spell checker issues
Mads Kiilerich <madski@unity3d.com>
parents: 23097
diff changeset
  2697
        If size parameter is omitted, read everything"""
26478
a3f7e5461dbd util.chunkbuffer: special case reading everything
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26450
diff changeset
  2698
        if l is None:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2699
            return b''.join(self.iter)
26478
a3f7e5461dbd util.chunkbuffer: special case reading everything
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26450
diff changeset
  2700
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
  2701
        left = l
17962
4c29668ca316 util: make chunkbuffer non-quadratic on Windows
Matt Mackall <mpm@selenic.com>
parents: 17560
diff changeset
  2702
        buf = []
16873
37e081609828 util: simplify queue management in chunkbuffer
Bryan O'Sullivan <bryano@fb.com>
parents: 16834
diff changeset
  2703
        queue = self._queue
26478
a3f7e5461dbd util.chunkbuffer: special case reading everything
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26450
diff changeset
  2704
        while left > 0:
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
  2705
            # refill the queue
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
  2706
            if not queue:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2707
                target = 2 ** 18
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
  2708
                for chunk in self.iter:
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
  2709
                    queue.append(chunk)
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
  2710
                    target -= len(chunk)
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
  2711
                    if target <= 0:
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
  2712
                        break
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
  2713
                if not queue:
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
  2714
                    break
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
  2715
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2716
            # The easy way to do this would be to queue.popleft(), modify the
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2717
            # chunk (if necessary), then queue.appendleft(). However, for cases
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2718
            # where we read partial chunk content, this incurs 2 dequeue
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2719
            # mutations and creates a new str for the remaining chunk in the
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2720
            # queue. Our code below avoids this overhead.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2721
26479
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
  2722
            chunk = queue[0]
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
  2723
            chunkl = len(chunk)
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2724
            offset = self._chunkoffset
26479
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
  2725
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
  2726
            # Use full chunk.
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2727
            if offset == 0 and left >= chunkl:
26479
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
  2728
                left -= chunkl
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
  2729
                queue.popleft()
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
  2730
                buf.append(chunk)
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2731
                # self._chunkoffset remains at 0.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2732
                continue
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2733
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2734
            chunkremaining = chunkl - offset
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2735
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2736
            # Use all of unconsumed part of chunk.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2737
            if left >= chunkremaining:
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2738
                left -= chunkremaining
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2739
                queue.popleft()
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2740
                # offset == 0 is enabled by block above, so this won't merely
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2741
                # copy via ``chunk[0:]``.
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2742
                buf.append(chunk[offset:])
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2743
                self._chunkoffset = 0
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2744
26479
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
  2745
            # Partial chunk needed.
46143f31290e util.chunkbuffer: refactor chunk handling logic
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26478
diff changeset
  2746
            else:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2747
                buf.append(chunk[offset : offset + left])
26480
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2748
                self._chunkoffset += left
6ae14d1ca3aa util.chunkbuffer: avoid extra mutations when reading partial chunks
Gregory Szorc <gregory.szorc@gmail.com>
parents: 26479
diff changeset
  2749
                left -= chunkremaining
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
  2750
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2751
        return b''.join(buf)
11758
a79214972da2 chunkbuffer: use += rather than cStringIO to reduce memory footprint
Matt Mackall <mpm@selenic.com>
parents: 11469
diff changeset
  2752
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2753
30181
7356e6b1f5b8 util: increase filechunkiter size to 128k
Mads Kiilerich <madski@unity3d.com>
parents: 30087
diff changeset
  2754
def filechunkiter(f, size=131072, limit=None):
2462
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
  2755
    """Create a generator that produces the data in the file size
30181
7356e6b1f5b8 util: increase filechunkiter size to 128k
Mads Kiilerich <madski@unity3d.com>
parents: 30087
diff changeset
  2756
    (default 131072) bytes at a time, up to optional limit (default is
2462
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
  2757
    to read all data).  Chunks may be less than size bytes if the
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
  2758
    chunk is the last chunk in the file, or the file is a socket or
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
  2759
    some other type of file that sometimes reads less data than is
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
  2760
    requested."""
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
  2761
    assert size >= 0
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
  2762
    assert limit is None or limit >= 0
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
  2763
    while True:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
  2764
        if limit is None:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
  2765
            nbytes = size
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
  2766
        else:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
  2767
            nbytes = min(limit, size)
2462
d610bcfd66a8 util: add limit to amount filechunkiter will read
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2448
diff changeset
  2768
        s = nbytes and f.read(nbytes)
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
  2769
        if not s:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
  2770
            break
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
  2771
        if limit:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 10264
diff changeset
  2772
            limit -= len(s)
1199
78ceaf83f28f Created a class in util called chunkbuffer that buffers reads from an
Eric Hopper <hopper@omnifarious.org>
parents: 1169
diff changeset
  2773
        yield s
1320
5f277e73778f Fix up representation of dates in hgweb.
Bryan O'Sullivan <bos@serpentine.com>
parents: 1312
diff changeset
  2774
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2775
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
  2776
class cappedreader:
36362
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2777
    """A file object proxy that allows reading up to N bytes.
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2778
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2779
    Given a source file object, instances of this type allow reading up to
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2780
    N bytes from that source file object. Attempts to read past the allowed
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2781
    limit are treated as EOF.
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2782
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2783
    It is assumed that I/O is not performed on the original file object
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2784
    in addition to I/O that is performed by this instance. If there is,
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2785
    state tracking will get out of sync and unexpected results will ensue.
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2786
    """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2787
36362
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2788
    def __init__(self, fh, limit):
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2789
        """Allow reading up to <limit> bytes from <fh>."""
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2790
        self._fh = fh
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2791
        self._left = limit
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2792
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2793
    def read(self, n=-1):
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2794
        if not self._left:
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2795
            return b''
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2796
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2797
        if n < 0:
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2798
            n = self._left
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2799
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2800
        data = self._fh.read(min(n, self._left))
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2801
        self._left -= len(data)
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2802
        assert self._left >= 0
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2803
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2804
        return data
01e29e885600 util: add a file object proxy that can read at most N bytes
Gregory Szorc <gregory.szorc@gmail.com>
parents: 36266
diff changeset
  2805
37052
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37044
diff changeset
  2806
    def readinto(self, b):
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37044
diff changeset
  2807
        res = self.read(len(b))
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37044
diff changeset
  2808
        if res is None:
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37044
diff changeset
  2809
            return None
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37044
diff changeset
  2810
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2811
        b[0 : len(res)] = res
37052
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37044
diff changeset
  2812
        return len(res)
8c3c47362934 wireproto: implement basic frame reading and processing
Gregory Szorc <gregory.szorc@gmail.com>
parents: 37044
diff changeset
  2813
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2814
18735
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
  2815
def unitcountfn(*unittable):
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
  2816
    '''return a function that renders a readable count of some quantity'''
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
  2817
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
  2818
    def go(count):
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
  2819
        for multiplier, divisor, format in unittable:
31946
f3b80537a70d util: fix human-readable printing of negative byte counts
Gábor Stefanik <gabor.stefanik@nng.com>
parents: 31934
diff changeset
  2820
            if abs(count) >= divisor * multiplier:
18735
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
  2821
                return format % (count / float(divisor))
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
  2822
        return unittable[-1][2] % count
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
  2823
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
  2824
    return go
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
  2825
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2826
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2827
def processlinerange(fromline: int, toline: int) -> Tuple[int, int]:
31662
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
  2828
    """Check that linerange <fromline>:<toline> makes sense and return a
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
  2829
    0-based range.
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
  2830
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
  2831
    >>> processlinerange(10, 20)
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
  2832
    (9, 20)
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
  2833
    >>> processlinerange(2, 1)
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
  2834
    Traceback (most recent call last):
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
  2835
        ...
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
  2836
    ParseError: line range must be positive
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
  2837
    >>> processlinerange(0, 5)
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
  2838
    Traceback (most recent call last):
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
  2839
        ...
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
  2840
    ParseError: fromline must be strictly positive
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
  2841
    """
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
  2842
    if toline - fromline < 0:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2843
        raise error.ParseError(_(b"line range must be positive"))
31662
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
  2844
    if fromline < 1:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2845
        raise error.ParseError(_(b"fromline must be strictly positive"))
31662
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
  2846
    return fromline - 1, toline
080734cd2440 revset: factor out linerange processing into a utility function
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 31598
diff changeset
  2847
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2848
18735
716cad930691 util: generalize bytecount to unitcountfn
Bryan O'Sullivan <bryano@fb.com>
parents: 18678
diff changeset
  2849
bytecount = unitcountfn(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2850
    (100, 1 << 30, _(b'%.0f GB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2851
    (10, 1 << 30, _(b'%.1f GB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2852
    (1, 1 << 30, _(b'%.2f GB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2853
    (100, 1 << 20, _(b'%.0f MB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2854
    (10, 1 << 20, _(b'%.1f MB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2855
    (1, 1 << 20, _(b'%.2f MB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2856
    (100, 1 << 10, _(b'%.0f KB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2857
    (10, 1 << 10, _(b'%.1f KB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2858
    (1, 1 << 10, _(b'%.2f KB')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2859
    (1, 1, _(b'%.0f bytes')),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2860
)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2861
16397
f0f7f3fab315 util: create bytecount array just once
Matt Mackall <mpm@selenic.com>
parents: 16383
diff changeset
  2862
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
  2863
class transformingwriter:
36837
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2864
    """Writable file wrapper to transform data by function"""
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2865
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2866
    def __init__(self, fp, encode):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2867
        self._fp = fp
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2868
        self._encode = encode
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2869
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2870
    def close(self):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2871
        self._fp.close()
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2872
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2873
    def flush(self):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2874
        self._fp.flush()
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2875
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2876
    def write(self, data):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2877
        return self._fp.write(self._encode(data))
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2878
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2879
31776
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31720
diff changeset
  2880
# Matches a single EOL which can either be a CRLF where repeated CR
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31720
diff changeset
  2881
# are removed or a LF. We do not care about old Macintosh files, so a
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31720
diff changeset
  2882
# stray CR is an error.
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31720
diff changeset
  2883
_eolre = remod.compile(br'\r*\n')
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31720
diff changeset
  2884
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2885
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2886
def tolf(s: bytes) -> bytes:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2887
    return _eolre.sub(b'\n', s)
31776
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31720
diff changeset
  2888
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2889
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2890
def tocrlf(s: bytes) -> bytes:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2891
    return _eolre.sub(b'\r\n', s)
31776
fe9b33bcec6a util: extract pure tolf/tocrlf() functions from eol extension
Yuya Nishihara <yuya@tcha.org>
parents: 31720
diff changeset
  2892
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2893
36837
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2894
def _crlfwriter(fp):
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2895
    return transformingwriter(fp, tocrlf)
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2896
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2897
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2898
if pycompat.oslinesep == b'\r\n':
31777
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31776
diff changeset
  2899
    tonativeeol = tocrlf
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31776
diff changeset
  2900
    fromnativeeol = tolf
36837
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2901
    nativeeolwriter = _crlfwriter
31777
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31776
diff changeset
  2902
else:
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31776
diff changeset
  2903
    tonativeeol = pycompat.identity
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31776
diff changeset
  2904
    fromnativeeol = pycompat.identity
36837
472c68cda3f8 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org>
parents: 36835
diff changeset
  2905
    nativeeolwriter = pycompat.identity
31777
6a5b69b0abec util: add helper to convert between LF and native EOL
Yuya Nishihara <yuya@tcha.org>
parents: 31776
diff changeset
  2906
48940
2974cdda819b util: remove iterfile() variant for buggy EINTR handling
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48913
diff changeset
  2907
2974cdda819b util: remove iterfile() variant for buggy EINTR handling
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48913
diff changeset
  2908
# TODO delete since workaround variant for Python 2 no longer needed.
2974cdda819b util: remove iterfile() variant for buggy EINTR handling
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48913
diff changeset
  2909
def iterfile(fp):
2974cdda819b util: remove iterfile() variant for buggy EINTR handling
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48913
diff changeset
  2910
    return fp
30395
10514a92860e util: add iterfile to workaround a fileobj.__iter__ issue with EINTR
Jun Wu <quark@fb.com>
parents: 30359
diff changeset
  2911
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2912
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2913
def iterlines(iterator: Iterable[bytes]) -> Iterator[bytes]:
7879
5c4026a289a4 templater: ability to display diffstat for log-like commands
Alexander Solovyov <piranha at piranha.org.ua>
parents: 7875
diff changeset
  2914
    for chunk in iterator:
5c4026a289a4 templater: ability to display diffstat for log-like commands
Alexander Solovyov <piranha at piranha.org.ua>
parents: 7875
diff changeset
  2915
        for line in chunk.splitlines():
5c4026a289a4 templater: ability to display diffstat for log-like commands
Alexander Solovyov <piranha at piranha.org.ua>
parents: 7875
diff changeset
  2916
            yield line
9610
d78fe60f6bda make path expanding more consistent
Alexander Solovyov <piranha@piranha.org.ua>
parents: 9569
diff changeset
  2917
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2918
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  2919
def expandpath(path: bytes) -> bytes:
9610
d78fe60f6bda make path expanding more consistent
Alexander Solovyov <piranha@piranha.org.ua>
parents: 9569
diff changeset
  2920
    return os.path.expanduser(os.path.expandvars(path))
10239
8e4be44a676f Find right hg command for detached process
Patrick Mezard <pmezard@gmail.com>
parents: 10199
diff changeset
  2921
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2922
13392
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
  2923
def interpolate(prefix, mapping, s, fn=None, escape_prefix=False):
11988
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
  2924
    """Return the result of interpolating items in the mapping into string s.
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
  2925
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
  2926
    prefix is a single character string, or a two character string with
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
  2927
    a backslash as the first character if the prefix needs to be escaped in
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
  2928
    a regular expression.
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
  2929
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
  2930
    fn is an optional function that will be applied to the replacement text
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
  2931
    just before replacement.
13392
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
  2932
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
  2933
    escape_prefix is an optional flag that allows using doubled prefix for
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
  2934
    its escaping.
11988
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
  2935
    """
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
  2936
    fn = fn or (lambda s: s)
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2937
    patterns = b'|'.join(mapping.keys())
13392
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
  2938
    if escape_prefix:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2939
        patterns += b'|' + prefix
13392
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
  2940
        if len(prefix) > 1:
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
  2941
            prefix_char = prefix[1:]
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
  2942
        else:
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
  2943
            prefix_char = prefix
777cef34a890 dispatch: support for $ escaping in shell-alias definition
Roman Sokolov <sokolov.r.v@gmail.com>
parents: 13375
diff changeset
  2944
        mapping[prefix_char] = prefix_char
35145
25c543944bc0 py3: add b'' to regular expressions which are raw strings
Pulkit Goyal <7895pulkit@gmail.com>
parents: 35014
diff changeset
  2945
    r = remod.compile(br'%s(%s)' % (prefix, patterns))
11988
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
  2946
    return r.sub(lambda x: fn(mapping[x.group()[1:]]), s)
8380ed691df8 util: add an interpolate() function to for replacing multiple values
Steve Losh <steve@stevelosh.com>
parents: 11946
diff changeset
  2947
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2948
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
  2949
timecount = unitcountfn(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2950
    (1, 1e3, _(b'%.0f s')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2951
    (100, 1, _(b'%.1f s')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2952
    (10, 1, _(b'%.2f s')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2953
    (1, 1, _(b'%.3f s')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2954
    (100, 0.001, _(b'%.1f ms')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2955
    (10, 0.001, _(b'%.2f ms')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2956
    (1, 0.001, _(b'%.3f ms')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2957
    (100, 0.000001, _(b'%.1f us')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2958
    (10, 0.000001, _(b'%.2f us')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2959
    (1, 0.000001, _(b'%.3f us')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2960
    (100, 0.000000001, _(b'%.1f ns')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2961
    (10, 0.000000001, _(b'%.2f ns')),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2962
    (1, 0.000000001, _(b'%.3f ns')),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2963
)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2964
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
  2965
38797
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2966
@attr.s
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
  2967
class timedcmstats:
38797
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2968
    """Stats information produced by the timedcm context manager on entering."""
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2969
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2970
    # the starting value of the timer as a float (meaning and resulution is
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2971
    # platform dependent, see util.timer)
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2972
    start = attr.ib(default=attr.Factory(lambda: timer()))
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2973
    # the number of seconds as a floating point value; starts at 0, updated when
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2974
    # the context is exited.
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2975
    elapsed = attr.ib(default=0)
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2976
    # the number of nested timedcm context managers.
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2977
    level = attr.ib(default=1)
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2978
38812
9d49bb117dde util: make new timedcmstats class Python 3 compatible
Martijn Pieters <mj@zopatista.com>
parents: 38797
diff changeset
  2979
    def __bytes__(self):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  2980
        return timecount(self.elapsed) if self.elapsed else b'<unknown>'
38797
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2981
38812
9d49bb117dde util: make new timedcmstats class Python 3 compatible
Martijn Pieters <mj@zopatista.com>
parents: 38797
diff changeset
  2982
    __str__ = encoding.strmethod(__bytes__)
9d49bb117dde util: make new timedcmstats class Python 3 compatible
Martijn Pieters <mj@zopatista.com>
parents: 38797
diff changeset
  2983
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  2984
38797
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2985
@contextlib.contextmanager
39259
e00123f63410 util: make timedcm require the label (API)
Augie Fackler <augie@google.com>
parents: 39258
diff changeset
  2986
def timedcm(whencefmt, *whenceargs):
38797
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2987
    """A context manager that produces timing information for a given context.
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2988
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2989
    On entering a timedcmstats instance is produced.
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2990
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2991
    This context manager is reentrant.
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2992
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2993
    """
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2994
    # track nested context managers
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2995
    timedcm._nested += 1
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2996
    timing_stats = timedcmstats(level=timedcm._nested)
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  2997
    try:
39257
497effb0a04a util: make timedcm context manager also emit trace events
Augie Fackler <augie@google.com>
parents: 39209
diff changeset
  2998
        with tracing.log(whencefmt, *whenceargs):
497effb0a04a util: make timedcm context manager also emit trace events
Augie Fackler <augie@google.com>
parents: 39209
diff changeset
  2999
            yield timing_stats
38797
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  3000
    finally:
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  3001
        timing_stats.elapsed = timer() - timing_stats.start
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  3002
        timedcm._nested -= 1
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  3003
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3004
38797
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  3005
timedcm._nested = 0
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
  3006
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3007
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
  3008
def timed(func):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  3009
    """Report the execution time of a function call to stderr.
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
  3010
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
  3011
    During development, use as a decorator when you need to measure
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
  3012
    the cost of a function, e.g. as follows:
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
  3013
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
  3014
    @util.timed
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
  3015
    def foo(a, b, c):
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
  3016
        pass
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  3017
    """
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
  3018
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
  3019
    def wrapper(*args, **kwargs):
39258
331ab85e910b cleanup: make all uses of timedcm specify what they're timing
Augie Fackler <augie@google.com>
parents: 39257
diff changeset
  3020
        with timedcm(pycompat.bytestr(func.__name__)) as time_stats:
38797
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  3021
            result = func(*args, **kwargs)
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  3022
        stderr = procutil.stderr
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3023
        stderr.write(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3024
            b'%s%s: %s\n'
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3025
            % (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3026
                b' ' * time_stats.level * 2,
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3027
                pycompat.bytestr(func.__name__),
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3028
                time_stats,
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3029
            )
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3030
        )
38797
8751d1e2a7ff util: create a context manager to handle timing
Martijn Pieters <mj@zopatista.com>
parents: 38713
diff changeset
  3031
        return result
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3032
18736
af9ddea2cb99 util: add a timed function for use during development
Bryan O'Sullivan <bryano@fb.com>
parents: 18735
diff changeset
  3033
    return wrapper
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
  3034
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3035
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3036
_sizeunits = (
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3037
    (b'm', 2 ** 20),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3038
    (b'k', 2 ** 10),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3039
    (b'g', 2 ** 30),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3040
    (b'kb', 2 ** 10),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3041
    (b'mb', 2 ** 20),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3042
    (b'gb', 2 ** 30),
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3043
    (b'b', 1),
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3044
)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3045
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
  3046
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  3047
def sizetoint(s: bytes) -> int:
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  3048
    """Convert a space specifier to a byte count.
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
  3049
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  3050
    >>> sizetoint(b'30')
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
  3051
    30
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  3052
    >>> sizetoint(b'2.2kb')
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
  3053
    2252
34131
0fa781320203 doctest: bulk-replace string literals with b'' for Python 3
Yuya Nishihara <yuya@tcha.org>
parents: 34084
diff changeset
  3054
    >>> sizetoint(b'6M')
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
  3055
    6291456
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  3056
    """
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
  3057
    t = s.strip().lower()
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
  3058
    try:
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
  3059
        for k, u in _sizeunits:
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
  3060
            if t.endswith(k):
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3061
                return int(float(t[: -len(k)]) * u)
19194
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
  3062
        return int(t)
1d08df65cd3c util: migrate fileset._sizetoint to util.sizetoint
Bryan O'Sullivan <bryano@fb.com>
parents: 18938
diff changeset
  3063
    except ValueError:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3064
        raise error.ParseError(_(b"couldn't parse size: %s") % s)
19211
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
  3065
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3066
48946
642e31cb55f0 py3: use class X: instead of class X(object):
Gregory Szorc <gregory.szorc@gmail.com>
parents: 48940
diff changeset
  3067
class hooks:
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  3068
    """A collection of hook functions that can be used to extend a
26098
ce26928cbe41 spelling: behaviour -> behavior
timeless@mozdev.org
parents: 25672
diff changeset
  3069
    function's behavior. Hooks are called in lexicographic order,
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  3070
    based on the names of their sources."""
19211
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
  3071
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
  3072
    def __init__(self):
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
  3073
        self._hooks = []
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
  3074
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
  3075
    def add(self, source, hook):
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
  3076
        self._hooks.append((source, hook))
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
  3077
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
  3078
    def __call__(self, *args):
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
  3079
        self._hooks.sort(key=lambda x: x[0])
21046
cc13addbd62b util: enable "hooks" to return list of the values returned from each hooks
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 21018
diff changeset
  3080
        results = []
19211
3bfd7f1e7485 summary: augment output with info from extensions
Bryan O'Sullivan <bryano@fb.com>
parents: 19194
diff changeset
  3081
        for source, hook in self._hooks:
21046
cc13addbd62b util: enable "hooks" to return list of the values returned from each hooks
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 21018
diff changeset
  3082
            results.append(hook(*args))
cc13addbd62b util: enable "hooks" to return list of the values returned from each hooks
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents: 21018
diff changeset
  3083
        return results
20244
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
  3084
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3085
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3086
def getstackframes(skip=0, line=b' %-*s in %s\n', fileline=b'%s:%d', depth=0):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  3087
    """Yields lines for a nicely formatted stacktrace.
31315
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31314
diff changeset
  3088
    Skips the 'skip' last entries, then return the last 'depth' entries.
28497
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
  3089
    Each file+linenumber is formatted according to fileline.
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
  3090
    Each line is formatted according to line.
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
  3091
    If line is None, it yields:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
  3092
      length of longest filepath+line number,
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
  3093
      filepath+linenumber,
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
  3094
      function
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
  3095
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
  3096
    Not be used in production code but very convenient while developing.
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  3097
    """
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3098
    entries = [
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3099
        (fileline % (pycompat.sysbytes(fn), ln), pycompat.sysbytes(func))
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3100
        for fn, ln, func, _text in traceback.extract_stack()[: -skip - 1]
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3101
    ][-depth:]
28497
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
  3102
    if entries:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
  3103
        fnmax = max(len(entry[0]) for entry in entries)
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
  3104
        for fnln, func in entries:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
  3105
            if line is None:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
  3106
                yield (fnmax, fnln, func)
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
  3107
            else:
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
  3108
                yield line % (fnmax, fnln, func)
906fece80cfa util: refactor getstackframes
timeless <timeless@mozdev.org>
parents: 28496
diff changeset
  3109
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3110
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3111
def debugstacktrace(
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3112
    msg=b'stacktrace',
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3113
    skip=0,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3114
    f=procutil.stderr,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3115
    otherf=procutil.stdout,
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3116
    depth=0,
43754
02ededbef627 util: add an optional `prefix` argument to debugstacktrace
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43735
diff changeset
  3117
    prefix=b'',
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3118
):
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  3119
    """Writes a message to f (stderr) with a nicely formatted stacktrace.
31315
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31314
diff changeset
  3120
    Skips the 'skip' entries closest to the call, then show 'depth' entries.
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31314
diff changeset
  3121
    By default it will flush stdout first.
28496
b592564a803c util: reword debugstacktrace comment
timeless <timeless@mozdev.org>
parents: 28027
diff changeset
  3122
    It can be used everywhere and intentionally does not require an ui object.
20244
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
  3123
    Not be used in production code but very convenient while developing.
45942
89a2afe31e82 formating: upgrade to black 20.8b1
Augie Fackler <raf@durin42.com>
parents: 45146
diff changeset
  3124
    """
20542
be27652675ce util: debugstacktrace, flush before and after writing
Mads Kiilerich <madski@unity3d.com>
parents: 20353
diff changeset
  3125
    if otherf:
be27652675ce util: debugstacktrace, flush before and after writing
Mads Kiilerich <madski@unity3d.com>
parents: 20353
diff changeset
  3126
        otherf.flush()
43754
02ededbef627 util: add an optional `prefix` argument to debugstacktrace
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43735
diff changeset
  3127
    f.write(b'%s%s at:\n' % (prefix, msg.rstrip()))
31315
78ac7061f840 util: add debugstacktrace depth limit
Mads Kiilerich <madski@unity3d.com>
parents: 31314
diff changeset
  3128
    for line in getstackframes(skip + 1, depth=depth):
43754
02ededbef627 util: add an optional `prefix` argument to debugstacktrace
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 43735
diff changeset
  3129
        f.write(prefix + line)
20542
be27652675ce util: debugstacktrace, flush before and after writing
Mads Kiilerich <madski@unity3d.com>
parents: 20353
diff changeset
  3130
    f.flush()
20244
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
  3131
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3132
20244
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
  3133
# convenient shortcut
47d0843647d1 util: introduce util.debugstacktrace for showing a stack trace without crashing
Mads Kiilerich <madski@unity3d.com>
parents: 20202
diff changeset
  3134
dst = debugstacktrace
34554
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3135
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3136
34554
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3137
def safename(f, tag, ctx, others=None):
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3138
    """
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3139
    Generate a name that it is safe to rename f to in the given context.
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3140
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3141
    f:      filename to rename
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3142
    tag:    a string tag that will be included in the new name
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3143
    ctx:    a context, in which the new name must not exist
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3144
    others: a set of other filenames that the new name must not be in
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3145
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3146
    Returns a file name of the form oldname~tag[~number] which does not exist
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3147
    in the provided context and is not in the set of other names.
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3148
    """
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3149
    if others is None:
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3150
        others = set()
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3151
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3152
    fn = b'%s~%s' % (f, tag)
34554
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3153
    if fn not in ctx and fn not in others:
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3154
        return fn
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3155
    for n in itertools.count(1):
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3156
        fn = b'%s~%s~%s' % (f, tag, n)
34554
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3157
        if fn not in ctx and fn not in others:
6f11a74d489f util: add safename function for generating safe names to rename to
Mark Thomas <mbthomas@fb.com>
parents: 34467
diff changeset
  3158
            return fn
35754
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35582
diff changeset
  3159
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3160
35754
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35582
diff changeset
  3161
def readexactly(stream, n):
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35582
diff changeset
  3162
    '''read n bytes from stream.read and abort if less was available'''
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35582
diff changeset
  3163
    s = stream.read(n)
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35582
diff changeset
  3164
    if len(s) < n:
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3165
        raise error.Abort(
43117
8ff1ecfadcd1 cleanup: join string literals that are already on one line
Martin von Zweigbergk <martinvonz@google.com>
parents: 43106
diff changeset
  3166
            _(b"stream ended unexpectedly (got %d bytes, expected %d)")
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3167
            % (len(s), n)
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3168
        )
35754
fb0be099063f util: move 'readexactly' in the util module
Boris Feld <boris.feld@octobus.net>
parents: 35582
diff changeset
  3169
    return s
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3170
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3171
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3172
def uvarintencode(value):
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3173
    """Encode an unsigned integer value to a varint.
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3174
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3175
    A varint is a variable length integer of 1 or more bytes. Each byte
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3176
    except the last has the most significant bit set. The lower 7 bits of
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3177
    each byte store the 2's complement representation, least significant group
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3178
    first.
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3179
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3180
    >>> uvarintencode(0)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3181
    '\\x00'
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3182
    >>> uvarintencode(1)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3183
    '\\x01'
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3184
    >>> uvarintencode(127)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3185
    '\\x7f'
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3186
    >>> uvarintencode(1337)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3187
    '\\xb9\\n'
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3188
    >>> uvarintencode(65536)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3189
    '\\x80\\x80\\x04'
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3190
    >>> uvarintencode(-1)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3191
    Traceback (most recent call last):
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3192
        ...
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3193
    ProgrammingError: negative value for uvarint: -1
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3194
    """
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3195
    if value < 0:
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3196
        raise error.ProgrammingError(b'negative value for uvarint: %d' % value)
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3197
    bits = value & 0x7F
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3198
    value >>= 7
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3199
    bytes = []
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3200
    while value:
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3201
        bytes.append(pycompat.bytechr(0x80 | bits))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3202
        bits = value & 0x7F
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3203
        value >>= 7
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3204
    bytes.append(pycompat.bytechr(bits))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3205
43077
687b865b95ad formatting: byteify all mercurial/ and hgext/ string literals
Augie Fackler <augie@google.com>
parents: 43076
diff changeset
  3206
    return b''.join(bytes)
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3207
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3208
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3209
def uvarintdecodestream(fh):
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3210
    """Decode an unsigned variable length integer from a stream.
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3211
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3212
    The passed argument is anything that has a ``.read(N)`` method.
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3213
49813
54114bba7c7e tests: drop some obsolete py2 handling in util.py doctest
Matt Harbison <matt_harbison@yahoo.com>
parents: 49572
diff changeset
  3214
    >>> from io import BytesIO
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3215
    >>> uvarintdecodestream(BytesIO(b'\\x00'))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3216
    0
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3217
    >>> uvarintdecodestream(BytesIO(b'\\x01'))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3218
    1
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3219
    >>> uvarintdecodestream(BytesIO(b'\\x7f'))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3220
    127
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3221
    >>> uvarintdecodestream(BytesIO(b'\\xb9\\n'))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3222
    1337
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3223
    >>> uvarintdecodestream(BytesIO(b'\\x80\\x80\\x04'))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3224
    65536
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3225
    >>> uvarintdecodestream(BytesIO(b'\\x80'))
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3226
    Traceback (most recent call last):
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3227
        ...
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3228
    Abort: stream ended unexpectedly (got 0 bytes, expected 1)
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3229
    """
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3230
    result = 0
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3231
    shift = 0
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3232
    while True:
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3233
        byte = ord(readexactly(fh, 1))
43076
2372284d9457 formatting: blacken the codebase
Augie Fackler <augie@google.com>
parents: 42766
diff changeset
  3234
        result |= (byte & 0x7F) << shift
35755
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3235
        if not (byte & 0x80):
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3236
            return result
2384523cee4d util: implement varint functions
Gregory Szorc <gregory.szorc@gmail.com>
parents: 35754
diff changeset
  3237
        shift += 7
45011
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3238
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3239
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3240
# Passing the '' locale means that the locale should be set according to the
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3241
# user settings (environment variables).
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3242
# Python sometimes avoids setting the global locale settings. When interfacing
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3243
# with C code (e.g. the curses module or the Subversion bindings), the global
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3244
# locale settings must be initialized correctly. Python 2 does not initialize
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3245
# the global locale settings on interpreter startup. Python 3 sometimes
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3246
# initializes LC_CTYPE, but not consistently at least on Windows. Therefore we
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3247
# explicitly initialize it to get consistent behavior if it's not already
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3248
# initialized. Since CPython commit 177d921c8c03d30daa32994362023f777624b10d,
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3249
# LC_CTYPE is always initialized. If we require Python 3.8+, we should re-check
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3250
# if we can remove this code.
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3251
@contextlib.contextmanager
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3252
def with_lc_ctype():
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3253
    oldloc = locale.setlocale(locale.LC_CTYPE, None)
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3254
    if oldloc == 'C':
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3255
        try:
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3256
            try:
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3257
                locale.setlocale(locale.LC_CTYPE, '')
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3258
            except locale.Error:
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3259
                # The likely case is that the locale from the environment
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3260
                # variables is unknown.
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3261
                pass
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3262
            yield
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3263
        finally:
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3264
            locale.setlocale(locale.LC_CTYPE, oldloc)
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3265
    else:
1bab6b61b62b curses: do not initialize LC_ALL to user settings (issue6358)
Manuel Jacob <me@manueljacob.de>
parents: 44629
diff changeset
  3266
        yield
45061
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3267
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3268
51287
f15cb5111a1e pytype: move some type comment to proper annotation
Pierre-Yves David <pierre-yves.david@octobus.net>
parents: 51285
diff changeset
  3269
def _estimatememory() -> Optional[int]:
45061
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3270
    """Provide an estimate for the available system memory in Bytes.
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3271
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3272
    If no estimate can be provided on the platform, returns None.
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3273
    """
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3274
    if pycompat.sysplatform.startswith(b'win'):
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3275
        # On Windows, use the GlobalMemoryStatusEx kernel function directly.
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3276
        from ctypes import c_long as DWORD, c_ulonglong as DWORDLONG
46645
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46113
diff changeset
  3277
        from ctypes.wintypes import (  # pytype: disable=import-error
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46113
diff changeset
  3278
            Structure,
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46113
diff changeset
  3279
            byref,
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46113
diff changeset
  3280
            sizeof,
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46113
diff changeset
  3281
            windll,
7711853110b9 typing: add some type annotations to mercurial/util.py
Matt Harbison <matt_harbison@yahoo.com>
parents: 46113
diff changeset
  3282
        )
45061
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3283
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3284
        class MEMORYSTATUSEX(Structure):
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3285
            _fields_ = [
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3286
                ('dwLength', DWORD),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3287
                ('dwMemoryLoad', DWORD),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3288
                ('ullTotalPhys', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3289
                ('ullAvailPhys', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3290
                ('ullTotalPageFile', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3291
                ('ullAvailPageFile', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3292
                ('ullTotalVirtual', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3293
                ('ullAvailVirtual', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3294
                ('ullExtendedVirtual', DWORDLONG),
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3295
            ]
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3296
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3297
        x = MEMORYSTATUSEX()
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3298
        x.dwLength = sizeof(x)
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3299
        windll.kernel32.GlobalMemoryStatusEx(byref(x))
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3300
        return x.ullAvailPhys
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3301
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3302
    # On newer Unix-like systems and Mac OSX, the sysconf interface
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3303
    # can be used. _SC_PAGE_SIZE is part of POSIX; _SC_PHYS_PAGES
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3304
    # seems to be implemented on most systems.
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3305
    try:
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3306
        pagesize = os.sysconf(os.sysconf_names['SC_PAGE_SIZE'])
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3307
        pages = os.sysconf(os.sysconf_names['SC_PHYS_PAGES'])
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3308
        return pagesize * pages
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3309
    except OSError:  # sysconf can fail
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3310
        pass
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3311
    except KeyError:  # unknown parameter
02b17231f6c3 util: provide a helper function to estimate RAM size
Joerg Sonnenberger <joerg@bec.de>
parents: 45019
diff changeset
  3312
        pass