convert: when getting file from Perforce concatenate data at the end
As it turned out, even when getting relatively small files, concatenating
string data every time when new chunk is received is very inefficient.
Maintaining a string list of data chunks and concatenating everything in one go
at the end seems much more efficient - in my testing it made getting 40 MB file
7 times faster, whilst converting of a particularly big changelist with some big
files went down from 20 hours to 3 hours.
--- a/hgext/convert/p4.py Sat Jul 18 17:10:28 2015 -0700
+++ b/hgext/convert/p4.py Thu Jul 30 00:58:05 2015 +0100
@@ -216,7 +216,7 @@
stdout = util.popen(cmd, mode='rb')
mode = None
- contents = ""
+ contents = []
keywords = None
for d in loaditer(stdout):
@@ -252,7 +252,7 @@
keywords = self.re_keywords
elif code == "text" or code == "binary":
- contents += data
+ contents.append(data)
lasterror = None
@@ -262,6 +262,8 @@
if mode is None:
return None, None
+ contents = ''.join(contents)
+
if keywords:
contents = keywords.sub("$\\1$", contents)
if mode == "l" and contents.endswith("\n"):