Augie Fackler <augie@google.com> [Thu, 10 Nov 2016 16:07:24 -0500] rev 30420
run-tests: forward Python USER_BASE from site (issue5425)
We do this so that any linters installed via pip install --user don't
break. See https://docs.python.org/2/library/site.html#site.USER_BASE
for a description of what this nonsense is all about.
An alternative would be to not set HOME, but that'll cause other
problems (see issue2707), or to forward every single path entry from
sys.path in PYTHONPATH (which seems sketchy in its own way).
Mads Kiilerich <madski@unity3d.com> [Mon, 14 Nov 2016 22:43:25 +0100] rev 30419
shelve: add missing space in help text
The change is trivial and unlikely to have been translated so we update
translation files too.
Jun Wu <quark@fb.com> [Tue, 15 Nov 2016 20:25:51 +0000] rev 30418
util: improve iterfile so it chooses code path wisely
We have performance concerns on "iterfile" as it is 4X slower on normal
files. While modern systems have the nice property that reading a "fast"
(on-disk) file cannot be interrupted and should be made use of.
This patch dumps the related knowledge in comments. And "iterfile" chooses
code paths wisely:
1. If it's CPython 3, or PyPY, use the fast path.
2. If fp is a normal file, use the fast path.
3. If fp is not a normal file and CPython version >= 2.7.4, use the same
workaround (4x slower) as before.
4. If fp is not a normal file and CPython version < 2.7.4, use another
workaround (2x slower but may block longer then necessary) which
basically re-invents the buffer + readline logic in Python.
This will give us good confidence on both correctness and performance
dealing with EINTR in iterfile(fp) for all known supported Python versions.
Augie Fackler <augie@google.com> [Wed, 16 Nov 2016 23:29:28 -0500] rev 30417
merge with stable
Jun Wu <quark@fb.com> [Sat, 12 Nov 2016 03:06:07 +0000] rev 30416
worker: stop using a separate thread waiting for children
Now that we have a SIGCHLD hander, and it could get executed when waiting
for I/O. It's no longer necessary to have a separated waitpid thread. So
just remove it.
Jun Wu <quark@fb.com> [Sat, 12 Nov 2016 03:07:22 +0000] rev 30415
worker: add a SIGCHLD handler to collect worker immediately
As planned by previous patches, add a SIGCHLD handler to get notifications
about worker exits, and deals with worker failure immediately.
Note that the SIGCHLD handler gets unregistered before killworkers(), so
SIGCHLD won't interrupt "killworkers" - making it harder to send kill
signals to waited processes.
Jun Wu <quark@fb.com> [Tue, 15 Nov 2016 02:12:16 +0000] rev 30414
worker: make waitforworkers reentrant
We are going to use it in the SIGCHLD handler. The handler will be executed
in the main thread with the non-blocking version of waitpid, while the
waitforworkers thread runs the blocking version. It's possible that one of
them collects a worker and makes the other error out (no child to wait).
This patch handles these errors: ECHILD is ignored. EINTR needs a retry.
The "pids" set is designed to be only modifiable by "waitforworkers". And we
only remove items after a successful waitpid. Since a child process can only
be "waitpid"-ed once. It's guaranteed that "pids.remove(p)" won't be called
with duplicated "p"s. And once a "p" is removed from "pids", that "p" does
not need to be killed or waited any more.
Jun Wu <quark@fb.com> [Tue, 15 Nov 2016 02:10:40 +0000] rev 30413
worker: change "pids" to a set
There is no need to keep any order of the "pids" array. A set is more
efficient for the "remove" operation. And the following patch will use that.
Jun Wu <quark@fb.com> [Thu, 28 Jul 2016 20:57:07 +0100] rev 30412
worker: allow waitforworkers to be non-blocking
This patch adds a boolean flag to waitforworkers and makes it non-blocking
if set to True.
This is to make it possible that we can reap our workers while keep other
unrelated children untouched, after receiving SIGCHLD.
Jun Wu <quark@fb.com> [Thu, 28 Jul 2016 20:51:20 +0100] rev 30411
worker: wait worker pid explicitly
Before this patch, waitforworkers uses os.wait() to collect child workers, and
only wait len(pids) processes. This can have serious issues if other code
spawns new processes and does not reap them: 1. worker.py may get wrong exit
code and kill innocent workers. 2. worker.py may continue without waiting for
all workers to complete.
This patch fixes the issue by using waitpid to wait worker pid explicitly.
However, this patch introduces a new issue: worker failure may not be handled
immediately. The issue will be addressed in next patches.