mercurial/keepalive.py
author Mike Hommey <mh@glandium.org>
Wed, 24 Sep 2014 16:00:47 +0900
changeset 22958 bb7a911b138e
parent 22505 232d437af120
child 25660 328739ea70c3
permissions -rw-r--r--
util: move md5 back next to sha1 and allow to call it without an argument This effectively backs out changeset 908c5906091b. The API change is done so that both util.sha1 and util.md5 can be called the same way. The function is moved in order to use it for md5 checksumming for an upcoming bundle2 feature.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     1
#   This library is free software; you can redistribute it and/or
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     2
#   modify it under the terms of the GNU Lesser General Public
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     3
#   License as published by the Free Software Foundation; either
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     4
#   version 2.1 of the License, or (at your option) any later version.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     5
#
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     6
#   This library is distributed in the hope that it will be useful,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     7
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     8
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
     9
#   Lesser General Public License for more details.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    10
#
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    11
#   You should have received a copy of the GNU Lesser General Public
15782
7de7630053cb Remove FSF mailing address from GPL headers
Martin Geisler <mg@aragost.com>
parents: 14958
diff changeset
    12
#   License along with this library; if not, see
7de7630053cb Remove FSF mailing address from GPL headers
Martin Geisler <mg@aragost.com>
parents: 14958
diff changeset
    13
#   <http://www.gnu.org/licenses/>.
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    14
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    15
# This file is part of urlgrabber, a high-level cross-protocol url-grabber
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    16
# Copyright 2002-2004 Michael D. Stenner, Ryan Tomayko
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    17
4026
8520a773a141 fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2600
diff changeset
    18
# Modified by Benoit Boissinot:
8520a773a141 fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2600
diff changeset
    19
#  - fix for digest auth (inspired from urllib2.py @ Python v2.4)
6470
ac0bcd951c2c python 2.6 compatibility: compatibility wrappers for hash functions
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 6001
diff changeset
    20
# Modified by Dirkjan Ochtman:
ac0bcd951c2c python 2.6 compatibility: compatibility wrappers for hash functions
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 6001
diff changeset
    21
#  - import md5 function from a local util module
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
    22
# Modified by Augie Fackler:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
    23
#  - add safesend method and use it to prevent broken pipe errors
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
    24
#    on large POST requests
4026
8520a773a141 fix for digest auth when using keepalive.py
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 2600
diff changeset
    25
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    26
"""An HTTP handler for urllib2 that supports HTTP 1.1 and keepalive.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    27
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    28
>>> import urllib2
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    29
>>> from keepalive import HTTPHandler
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    30
>>> keepalive_handler = HTTPHandler()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    31
>>> opener = urllib2.build_opener(keepalive_handler)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    32
>>> urllib2.install_opener(opener)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
    33
>>>
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    34
>>> fo = urllib2.urlopen('http://www.python.org')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    35
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    36
If a connection to a given host is requested, and all of the existing
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    37
connections are still in use, another connection will be opened.  If
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    38
the handler tries to use an existing connection but it fails in some
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    39
way, it will be closed and removed from the pool.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    40
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    41
To remove the handler, simply re-run build_opener with no arguments, and
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    42
install that opener.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    43
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    44
You can explicitly close connections by using the close_connection()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    45
method of the returned file-like object (described below) or you can
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    46
use the handler methods:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    47
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    48
  close_connection(host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    49
  close_all()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    50
  open_connections()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    51
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    52
NOTE: using the close_connection and close_all methods of the handler
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    53
should be done with care when using multiple threads.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    54
  * there is nothing that prevents another thread from creating new
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    55
    connections immediately after connections are closed
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    56
  * no checks are done to prevent in-use connections from being closed
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    57
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    58
>>> keepalive_handler.close_all()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    59
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    60
EXTRA ATTRIBUTES AND METHODS
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    61
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    62
  Upon a status of 200, the object returned has a few additional
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    63
  attributes and methods, which should not be used if you want to
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    64
  remain consistent with the normal urllib2-returned objects:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    65
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    66
    close_connection()  -  close the connection to the host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    67
    readlines()         -  you know, readlines()
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 17251
diff changeset
    68
    status              -  the return status (i.e. 404)
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 17251
diff changeset
    69
    reason              -  english translation of status (i.e. 'File not found')
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    70
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    71
  If you want the best of both worlds, use this inside an
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    72
  AttributeError-catching try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    73
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    74
  >>> try: status = fo.status
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    75
  >>> except AttributeError: status = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    76
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    77
  Unfortunately, these are ONLY there if status == 200, so it's not
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    78
  easy to distinguish between non-200 responses.  The reason is that
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    79
  urllib2 tries to do clever things with error codes 301, 302, 401,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    80
  and 407, and it wraps the object upon return.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    81
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    82
  For python versions earlier than 2.4, you can avoid this fancy error
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    83
  handling by setting the module-level global HANDLE_ERRORS to zero.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    84
  You see, prior to 2.4, it's the HTTP Handler's job to determine what
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    85
  to handle specially, and what to just pass up.  HANDLE_ERRORS == 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    86
  means "pass everything up".  In python 2.4, however, this job no
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    87
  longer belongs to the HTTP Handler and is now done by a NEW handler,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    88
  HTTPErrorProcessor.  Here's the bottom line:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    89
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    90
    python version < 2.4
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    91
        HANDLE_ERRORS == 1  (default) pass up 200, treat the rest as
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    92
                            errors
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    93
        HANDLE_ERRORS == 0  pass everything up, error processing is
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    94
                            left to the calling code
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    95
    python version >= 2.4
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    96
        HANDLE_ERRORS == 1  pass up 200, treat the rest as errors
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    97
        HANDLE_ERRORS == 0  (default) pass everything up, let the
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    98
                            other handlers (specifically,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
    99
                            HTTPErrorProcessor) decide what to do
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   100
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   101
  In practice, setting the variable either way makes little difference
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   102
  in python 2.4, so for the most consistent behavior across versions,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   103
  you probably just want to use the defaults, which will give you
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   104
  exceptions on errors.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   105
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   106
"""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   107
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
   108
# $Id: keepalive.py,v 1.14 2006/04/04 21:00:32 mstenner Exp $
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   109
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   110
import errno
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   111
import httplib
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   112
import socket
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   113
import thread
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   114
import urllib2
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   115
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   116
DEBUG = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   117
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   118
import sys
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   119
if sys.version_info < (2, 4):
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   120
    HANDLE_ERRORS = 1
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   121
else: HANDLE_ERRORS = 0
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   122
14764
a7d5816087a9 classes: fix class style problems found by b071cd58af50
Thomas Arendsen Hein <thomas@intevation.de>
parents: 14494
diff changeset
   123
class ConnectionManager(object):
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   124
    """
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   125
    The connection manager must be able to:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   126
      * keep track of all existing
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   127
      """
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   128
    def __init__(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   129
        self._lock = thread.allocate_lock()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   130
        self._hostmap = {} # map hosts to a list of connections
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   131
        self._connmap = {} # map connections to host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   132
        self._readymap = {} # map connection to ready state
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   133
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   134
    def add(self, host, connection, ready):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   135
        self._lock.acquire()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   136
        try:
16686
67964cda8701 cleanup: "not x in y" -> "x not in y"
Brodie Rao <brodie@sf.io>
parents: 15782
diff changeset
   137
            if host not in self._hostmap:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   138
                self._hostmap[host] = []
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   139
            self._hostmap[host].append(connection)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   140
            self._connmap[connection] = host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   141
            self._readymap[connection] = ready
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   142
        finally:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   143
            self._lock.release()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   144
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   145
    def remove(self, connection):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   146
        self._lock.acquire()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   147
        try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   148
            try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   149
                host = self._connmap[connection]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   150
            except KeyError:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   151
                pass
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   152
            else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   153
                del self._connmap[connection]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   154
                del self._readymap[connection]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   155
                self._hostmap[host].remove(connection)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   156
                if not self._hostmap[host]: del self._hostmap[host]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   157
        finally:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   158
            self._lock.release()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   159
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   160
    def set_ready(self, connection, ready):
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   161
        try:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   162
            self._readymap[connection] = ready
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   163
        except KeyError:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   164
            pass
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   165
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   166
    def get_ready_conn(self, host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   167
        conn = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   168
        self._lock.acquire()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   169
        try:
5915
d0576d065993 Prefer i in d over d.has_key(i)
Christian Ebert <blacktrash@gmx.net>
parents: 4026
diff changeset
   170
            if host in self._hostmap:
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   171
                for c in self._hostmap[host]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   172
                    if self._readymap[c]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   173
                        self._readymap[c] = 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   174
                        conn = c
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   175
                        break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   176
        finally:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   177
            self._lock.release()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   178
        return conn
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   179
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   180
    def get_all(self, host=None):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   181
        if host:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   182
            return list(self._hostmap.get(host, []))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   183
        else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   184
            return dict(self._hostmap)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   185
14764
a7d5816087a9 classes: fix class style problems found by b071cd58af50
Thomas Arendsen Hein <thomas@intevation.de>
parents: 14494
diff changeset
   186
class KeepAliveHandler(object):
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   187
    def __init__(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   188
        self._cm = ConnectionManager()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   189
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   190
    #### Connection Management
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   191
    def open_connections(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   192
        """return a list of connected hosts and the number of connections
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   193
        to each.  [('foo.com:80', 2), ('bar.org', 1)]"""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   194
        return [(host, len(li)) for (host, li) in self._cm.get_all().items()]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   195
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   196
    def close_connection(self, host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   197
        """close connection(s) to <host>
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   198
        host is the host:port spec, as in 'www.cnn.com:8080' as passed in.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   199
        no error occurs if there is no connection to that host."""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   200
        for h in self._cm.get_all(host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   201
            self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   202
            h.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   203
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   204
    def close_all(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   205
        """close all open connections"""
7622
4dd7b28003d2 use dict.iteritems() rather than dict.items()
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 6470
diff changeset
   206
        for host, conns in self._cm.get_all().iteritems():
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   207
            for h in conns:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   208
                self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   209
                h.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   210
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   211
    def _request_closed(self, request, host, connection):
17251
98166640b356 help: fix some instances of 'the the'
Mads Kiilerich <mads@kiilerich.com>
parents: 16705
diff changeset
   212
        """tells us that this request is now closed and that the
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   213
        connection is ready for another request"""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   214
        self._cm.set_ready(connection, 1)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   215
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   216
    def _remove_connection(self, host, connection, close=0):
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   217
        if close:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   218
            connection.close()
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   219
        self._cm.remove(connection)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   220
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   221
    #### Transaction Execution
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   222
    def http_open(self, req):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   223
        return self.do_open(HTTPConnection, req)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   224
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   225
    def do_open(self, http_class, req):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   226
        host = req.get_host()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   227
        if not host:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   228
            raise urllib2.URLError('no host given')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   229
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   230
        try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   231
            h = self._cm.get_ready_conn(host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   232
            while h:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   233
                r = self._reuse_connection(h, req, host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   234
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   235
                # if this response is non-None, then it worked and we're
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   236
                # done.  Break out, skipping the else block.
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   237
                if r:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   238
                    break
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   239
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   240
                # connection is bad - possibly closed by server
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   241
                # discard it and ask for the next free connection
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   242
                h.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   243
                self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   244
                h = self._cm.get_ready_conn(host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   245
            else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   246
                # no (working) free connections were found.  Create a new one.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   247
                h = http_class(host)
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   248
                if DEBUG:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   249
                    DEBUG.info("creating new connection to %s (%d)",
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   250
                               host, id(h))
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   251
                self._cm.add(host, h, 0)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   252
                self._start_transaction(h, req)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   253
                r = h.getresponse()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   254
        except (socket.error, httplib.HTTPException), err:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   255
            raise urllib2.URLError(err)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   256
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   257
        # if not a persistent connection, don't try to reuse it
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   258
        if r.will_close:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   259
            self._cm.remove(h)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   260
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   261
        if DEBUG:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   262
            DEBUG.info("STATUS: %s, %s", r.status, r.reason)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   263
        r._handler = self
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   264
        r._host = host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   265
        r._url = req.get_full_url()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   266
        r._connection = h
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   267
        r.code = r.status
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
   268
        r.headers = r.msg
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
   269
        r.msg = r.reason
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   270
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   271
        if r.status == 200 or not HANDLE_ERRORS:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   272
            return r
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   273
        else:
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
   274
            return self.parent.error('http', req, r,
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
   275
                                     r.status, r.msg, r.headers)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   276
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   277
    def _reuse_connection(self, h, req, host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   278
        """start the transaction with a re-used connection
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   279
        return a response object (r) upon success or None on failure.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   280
        This DOES not close or remove bad connections in cases where
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   281
        it returns.  However, if an unexpected exception occurs, it
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   282
        will close and remove the connection before re-raising.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   283
        """
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   284
        try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   285
            self._start_transaction(h, req)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   286
            r = h.getresponse()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   287
            # note: just because we got something back doesn't mean it
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   288
            # worked.  We'll check the version below, too.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   289
        except (socket.error, httplib.HTTPException):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   290
            r = None
16705
c2d9ef43ff6c check-code: ignore naked excepts with a "re-raise" comment
Brodie Rao <brodie@sf.io>
parents: 16688
diff changeset
   291
        except: # re-raises
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   292
            # adding this block just in case we've missed
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   293
            # something we will still raise the exception, but
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   294
            # lets try and close the connection and remove it
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   295
            # first.  We previously got into a nasty loop
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   296
            # where an exception was uncaught, and so the
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   297
            # connection stayed open.  On the next try, the
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 17251
diff changeset
   298
            # same exception was raised, etc.  The trade-off is
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   299
            # that it's now possible this call will raise
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   300
            # a DIFFERENT exception
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   301
            if DEBUG:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   302
                DEBUG.error("unexpected exception - closing "
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   303
                            "connection to %s (%d)", host, id(h))
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   304
            self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   305
            h.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   306
            raise
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   307
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   308
        if r is None or r.version == 9:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   309
            # httplib falls back to assuming HTTP 0.9 if it gets a
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   310
            # bad header back.  This is most likely to happen if
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   311
            # the socket has been closed by the server since we
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   312
            # last used the connection.
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   313
            if DEBUG:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   314
                DEBUG.info("failed to re-use connection to %s (%d)",
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   315
                           host, id(h))
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   316
            r = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   317
        else:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   318
            if DEBUG:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   319
                DEBUG.info("re-using connection to %s (%d)", host, id(h))
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   320
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   321
        return r
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   322
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   323
    def _start_transaction(self, h, req):
8233
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   324
        # What follows mostly reimplements HTTPConnection.request()
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   325
        # except it adds self.parent.addheaders in the mix.
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   326
        headers = req.headers.copy()
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   327
        if sys.version_info >= (2, 4):
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   328
            headers.update(req.unredirected_hdrs)
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   329
        headers.update(self.parent.addheaders)
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   330
        headers = dict((n.lower(), v) for n, v in headers.items())
8233
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   331
        skipheaders = {}
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   332
        for n in ('host', 'accept-encoding'):
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   333
            if n in headers:
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   334
                skipheaders['skip_' + n.replace('-', '_')] = 1
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   335
        try:
8146
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   336
            if req.has_data():
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   337
                data = req.get_data()
8233
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   338
                h.putrequest('POST', req.get_selector(), **skipheaders)
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   339
                if 'content-type' not in headers:
8146
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   340
                    h.putheader('Content-type',
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   341
                                'application/x-www-form-urlencoded')
8233
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   342
                if 'content-length' not in headers:
8146
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   343
                    h.putheader('Content-length', '%d' % len(data))
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   344
            else:
8233
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   345
                h.putrequest('GET', req.get_selector(), **skipheaders)
8146
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   346
        except (socket.error), err:
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   347
            raise urllib2.URLError(err)
8233
655c435efe92 keepalive: fix 4f13ed6ee544, reintroduce unredirected_hdrs
Patrick Mezard <pmezard@gmail.com>
parents: 8146
diff changeset
   348
        for k, v in headers.items():
8146
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   349
            h.putheader(k, v)
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   350
        h.endheaders()
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   351
        if req.has_data():
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   352
            h.send(data)
4f13ed6ee544 keepalive: attempt to fix issue1003
Matt Mackall <mpm@selenic.com>
parents: 7875
diff changeset
   353
5983
6f1fcbc58efa httprepo: use separate handlers for HTTP and HTTPS
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4026
diff changeset
   354
class HTTPHandler(KeepAliveHandler, urllib2.HTTPHandler):
6f1fcbc58efa httprepo: use separate handlers for HTTP and HTTPS
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4026
diff changeset
   355
    pass
6f1fcbc58efa httprepo: use separate handlers for HTTP and HTTPS
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 4026
diff changeset
   356
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   357
class HTTPResponse(httplib.HTTPResponse):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   358
    # we need to subclass HTTPResponse in order to
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   359
    # 1) add readline() and readlines() methods
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   360
    # 2) add close_connection() methods
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   361
    # 3) add info() and geturl() methods
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   362
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   363
    # in order to add readline(), read must be modified to deal with a
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   364
    # buffer.  example: readline must read a buffer and then spit back
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   365
    # one line at a time.  The only real alternative is to read one
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   366
    # BYTE at a time (ick).  Once something has been read, it can't be
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   367
    # put back (ok, maybe it can, but that's even uglier than this),
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   368
    # so if you THEN do a normal read, you must first take stuff from
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   369
    # the buffer.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   370
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 17251
diff changeset
   371
    # the read method wraps the original to accommodate buffering,
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   372
    # although read() never adds to the buffer.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   373
    # Both readline and readlines have been stolen with almost no
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   374
    # modification from socket.py
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   375
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   376
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   377
    def __init__(self, sock, debuglevel=0, strict=0, method=None):
17700
5b1b0e4e6902 keepalive: drop python 2.2 legacy code
Bryan O'Sullivan <bryano@fb.com>
parents: 17424
diff changeset
   378
        httplib.HTTPResponse.__init__(self, sock, debuglevel, method)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   379
        self.fileno = sock.fileno
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   380
        self.code = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   381
        self._rbuf = ''
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   382
        self._rbufsize = 8096
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   383
        self._handler = None # inserted by the handler later
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   384
        self._host = None    # (same)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   385
        self._url = None     # (same)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   386
        self._connection = None # (same)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   387
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   388
    _raw_read = httplib.HTTPResponse.read
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   389
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   390
    def close(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   391
        if self.fp:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   392
            self.fp.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   393
            self.fp = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   394
            if self._handler:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   395
                self._handler._request_closed(self, self._host,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   396
                                              self._connection)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   397
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   398
    def close_connection(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   399
        self._handler._remove_connection(self._host, self._connection, close=1)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   400
        self.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   401
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   402
    def info(self):
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
   403
        return self.headers
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   404
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   405
    def geturl(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   406
        return self._url
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   407
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   408
    def read(self, amt=None):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   409
        # the _rbuf test is only in this first if for speed.  It's not
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   410
        # logically necessary
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   411
        if self._rbuf and not amt is None:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   412
            L = len(self._rbuf)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   413
            if amt > L:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   414
                amt -= L
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   415
            else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   416
                s = self._rbuf[:amt]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   417
                self._rbuf = self._rbuf[amt:]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   418
                return s
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   419
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   420
        s = self._rbuf + self._raw_read(amt)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   421
        self._rbuf = ''
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   422
        return s
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   423
7781
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   424
    # stolen from Python SVN #68532 to fix issue1088
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   425
    def _read_chunked(self, amt):
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   426
        chunk_left = self.chunk_left
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   427
        value = ''
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   428
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   429
        # XXX This accumulates chunks by repeated string concatenation,
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   430
        # which is not efficient as the number or size of chunks gets big.
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   431
        while True:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   432
            if chunk_left is None:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   433
                line = self.fp.readline()
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   434
                i = line.find(';')
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   435
                if i >= 0:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   436
                    line = line[:i] # strip chunk-extensions
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   437
                try:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   438
                    chunk_left = int(line, 16)
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   439
                except ValueError:
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 17251
diff changeset
   440
                    # close the connection as protocol synchronization is
7781
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   441
                    # probably lost
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   442
                    self.close()
7868
2b901f9b37aa keepalive: fix reference to IncompleteRead
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents: 7781
diff changeset
   443
                    raise httplib.IncompleteRead(value)
7781
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   444
                if chunk_left == 0:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   445
                    break
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   446
            if amt is None:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   447
                value += self._safe_read(chunk_left)
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   448
            elif amt < chunk_left:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   449
                value += self._safe_read(amt)
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   450
                self.chunk_left = chunk_left - amt
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   451
                return value
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   452
            elif amt == chunk_left:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   453
                value += self._safe_read(amt)
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   454
                self._safe_read(2)  # toss the CRLF at the end of the chunk
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   455
                self.chunk_left = None
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   456
                return value
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   457
            else:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   458
                value += self._safe_read(chunk_left)
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   459
                amt -= chunk_left
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   460
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   461
            # we read the whole chunk, get another
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   462
            self._safe_read(2)      # toss the CRLF at the end of the chunk
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   463
            chunk_left = None
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   464
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   465
        # read and discard trailer up to the CRLF terminator
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   466
        ### note: we shouldn't have any trailers!
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   467
        while True:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   468
            line = self.fp.readline()
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   469
            if not line:
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   470
                # a vanishingly small number of sites EOF without
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   471
                # sending the trailer
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   472
                break
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   473
            if line == '\r\n':
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   474
                break
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   475
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   476
        # we read everything; close the "file"
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   477
        self.close()
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   478
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   479
        return value
a45206455d85 keepalive: borrow code from newer httplib to patch ValueError (issue1088)
Matt Mackall <mpm@selenic.com>
parents: 7622
diff changeset
   480
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   481
    def readline(self, limit=-1):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   482
        i = self._rbuf.find('\n')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   483
        while i < 0 and not (0 < limit <= len(self._rbuf)):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   484
            new = self._raw_read(self._rbufsize)
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   485
            if not new:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   486
                break
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   487
            i = new.find('\n')
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   488
            if i >= 0:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   489
                i = i + len(self._rbuf)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   490
            self._rbuf = self._rbuf + new
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   491
        if i < 0:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   492
            i = len(self._rbuf)
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   493
        else:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   494
            i = i + 1
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   495
        if 0 <= limit < len(self._rbuf):
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   496
            i = limit
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   497
        data, self._rbuf = self._rbuf[:i], self._rbuf[i:]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   498
        return data
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   499
19872
681f7b9213a4 check-code: check for spaces around = for named parameters
Mads Kiilerich <madski@unity3d.com>
parents: 17700
diff changeset
   500
    def readlines(self, sizehint=0):
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   501
        total = 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   502
        list = []
14494
1ffeeb91c55d check-code: flag 0/1 used as constant Boolean expression
Martin Geisler <mg@lazybytes.net>
parents: 10394
diff changeset
   503
        while True:
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   504
            line = self.readline()
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   505
            if not line:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   506
                break
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   507
            list.append(line)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   508
            total += len(line)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   509
            if sizehint and total >= sizehint:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   510
                break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   511
        return list
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   512
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   513
def safesend(self, str):
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   514
    """Send `str' to the server.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   515
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   516
    Shamelessly ripped off from httplib to patch a bad behavior.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   517
    """
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   518
    # _broken_pipe_resp is an attribute we set in this function
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   519
    # if the socket is closed while we're sending data but
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   520
    # the server sent us a response before hanging up.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   521
    # In that case, we want to pretend to send the rest of the
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   522
    # outgoing data, and then let the user use getresponse()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   523
    # (which we wrap) to get this last response before
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   524
    # opening a new socket.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   525
    if getattr(self, '_broken_pipe_resp', None) is not None:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   526
        return
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   527
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   528
    if self.sock is None:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   529
        if self.auto_open:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   530
            self.connect()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   531
        else:
16687
e34106fa0dc3 cleanup: "raise SomeException()" -> "raise SomeException"
Brodie Rao <brodie@sf.io>
parents: 16686
diff changeset
   532
            raise httplib.NotConnected
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   533
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   534
    # send the data to the server. if we get a broken pipe, then close
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   535
    # the socket. we want to reconnect when somebody tries to send again.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   536
    #
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   537
    # NOTE: we DO propagate the error, though, because we cannot simply
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   538
    #       ignore the error... the caller will know if they can retry.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   539
    if self.debuglevel > 0:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   540
        print "send:", repr(str)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   541
    try:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   542
        blocksize = 8192
14958
fd246aefedd3 keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents: 14764
diff changeset
   543
        read = getattr(str, 'read', None)
fd246aefedd3 keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents: 14764
diff changeset
   544
        if read is not None:
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   545
            if self.debuglevel > 0:
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 17251
diff changeset
   546
                print "sending a read()able"
14958
fd246aefedd3 keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents: 14764
diff changeset
   547
            data = read(blocksize)
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   548
            while data:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   549
                self.sock.sendall(data)
14958
fd246aefedd3 keepalive: use safehasattr instead of hasattr
Augie Fackler <durin42@gmail.com>
parents: 14764
diff changeset
   550
                data = read(blocksize)
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   551
        else:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   552
            self.sock.sendall(str)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   553
    except socket.error, v:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   554
        reraise = True
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   555
        if v[0] == errno.EPIPE:      # Broken pipe
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   556
            if self._HTTPConnection__state == httplib._CS_REQ_SENT:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   557
                self._broken_pipe_resp = None
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   558
                self._broken_pipe_resp = self.getresponse()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   559
                reraise = False
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   560
            self.close()
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   561
        if reraise:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   562
            raise
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   563
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   564
def wrapgetresponse(cls):
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   565
    """Wraps getresponse in cls with a broken-pipe sane version.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   566
    """
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   567
    def safegetresponse(self):
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   568
        # In safesend() we might set the _broken_pipe_resp
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   569
        # attribute, in which case the socket has already
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   570
        # been closed and we just need to give them the response
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   571
        # back. Otherwise, we use the normal response path.
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   572
        r = getattr(self, '_broken_pipe_resp', None)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   573
        if r is not None:
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   574
            return r
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   575
        return cls.getresponse(self)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   576
    safegetresponse.__doc__ = cls.getresponse.__doc__
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   577
    return safegetresponse
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   578
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   579
class HTTPConnection(httplib.HTTPConnection):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   580
    # use the modified response class
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   581
    response_class = HTTPResponse
9726
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   582
    send = safesend
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   583
    getresponse = wrapgetresponse(httplib.HTTPConnection)
430e59ff3437 keepalive: handle broken pipes gracefully during large POSTs
Augie Fackler <durin42@gmail.com>
parents: 8296
diff changeset
   584
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   585
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   586
#########################################################################
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   587
#####   TEST FUNCTIONS
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   588
#########################################################################
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   589
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   590
def error_handler(url):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   591
    global HANDLE_ERRORS
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   592
    orig = HANDLE_ERRORS
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   593
    keepalive_handler = HTTPHandler()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   594
    opener = urllib2.build_opener(keepalive_handler)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   595
    urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   596
    pos = {0: 'off', 1: 'on'}
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   597
    for i in (0, 1):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   598
        print "  fancy error handling %s (HANDLE_ERRORS = %i)" % (pos[i], i)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   599
        HANDLE_ERRORS = i
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   600
        try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   601
            fo = urllib2.urlopen(url)
7874
d812029cda85 cleanup: drop variables for unused return values
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents: 7868
diff changeset
   602
            fo.read()
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   603
            fo.close()
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   604
            try:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   605
                status, reason = fo.status, fo.reason
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   606
            except AttributeError:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   607
                status, reason = None, None
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   608
        except IOError, e:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   609
            print "  EXCEPTION: %s" % e
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   610
            raise
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   611
        else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   612
            print "  status = %s, reason = %s" % (status, reason)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   613
    HANDLE_ERRORS = orig
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   614
    hosts = keepalive_handler.open_connections()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   615
    print "open connections:", hosts
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   616
    keepalive_handler.close_all()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   617
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   618
def continuity(url):
22958
bb7a911b138e util: move md5 back next to sha1 and allow to call it without an argument
Mike Hommey <mh@glandium.org>
parents: 22505
diff changeset
   619
    from util import md5
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   620
    format = '%25s: %s'
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   621
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   622
    # first fetch the file with the normal http handler
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   623
    opener = urllib2.build_opener()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   624
    urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   625
    fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   626
    foo = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   627
    fo.close()
22505
232d437af120 keepalive: fix how md5 is used
Mike Hommey <mh@glandium.org>
parents: 19872
diff changeset
   628
    m = md5(foo)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   629
    print format % ('normal urllib', m.hexdigest())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   630
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   631
    # now install the keepalive handler and try again
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   632
    opener = urllib2.build_opener(HTTPHandler())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   633
    urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   634
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   635
    fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   636
    foo = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   637
    fo.close()
22505
232d437af120 keepalive: fix how md5 is used
Mike Hommey <mh@glandium.org>
parents: 19872
diff changeset
   638
    m = md5(foo)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   639
    print format % ('keepalive read', m.hexdigest())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   640
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   641
    fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   642
    foo = ''
14494
1ffeeb91c55d check-code: flag 0/1 used as constant Boolean expression
Martin Geisler <mg@lazybytes.net>
parents: 10394
diff changeset
   643
    while True:
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   644
        f = fo.readline()
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   645
        if f:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   646
            foo = foo + f
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   647
        else: break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   648
    fo.close()
22505
232d437af120 keepalive: fix how md5 is used
Mike Hommey <mh@glandium.org>
parents: 19872
diff changeset
   649
    m = md5(foo)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   650
    print format % ('keepalive readline', m.hexdigest())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   651
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   652
def comp(N, url):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   653
    print '  making %i connections to:\n  %s' % (N, url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   654
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   655
    sys.stdout.write('  first using the normal urllib handlers')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   656
    # first use normal opener
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   657
    opener = urllib2.build_opener()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   658
    urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   659
    t1 = fetch(N, url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   660
    print '  TIME: %.3f s' % t1
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   661
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   662
    sys.stdout.write('  now using the keepalive handler       ')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   663
    # now install the keepalive handler and try again
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   664
    opener = urllib2.build_opener(HTTPHandler())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   665
    urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   666
    t2 = fetch(N, url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   667
    print '  TIME: %.3f s' % t2
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   668
    print '  improvement factor: %.2f' % (t1 / t2)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   669
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   670
def fetch(N, url, delay=0):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   671
    import time
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   672
    lens = []
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   673
    starttime = time.time()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   674
    for i in range(N):
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   675
        if delay and i > 0:
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   676
            time.sleep(delay)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   677
        fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   678
        foo = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   679
        fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   680
        lens.append(len(foo))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   681
    diff = time.time() - starttime
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   682
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   683
    j = 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   684
    for i in lens[1:]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   685
        j = j + 1
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   686
        if not i == lens[0]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   687
            print "WARNING: inconsistent length on read %i: %i" % (j, i)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   688
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   689
    return diff
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   690
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   691
def test_timeout(url):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   692
    global DEBUG
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   693
    dbbackup = DEBUG
14764
a7d5816087a9 classes: fix class style problems found by b071cd58af50
Thomas Arendsen Hein <thomas@intevation.de>
parents: 14494
diff changeset
   694
    class FakeLogger(object):
10282
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   695
        def debug(self, msg, *args):
08a0f04b56bd many, many trivial check-code fixups
Matt Mackall <mpm@selenic.com>
parents: 9726
diff changeset
   696
            print msg % args
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   697
        info = warning = error = debug
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   698
    DEBUG = FakeLogger()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   699
    print "  fetching the file to establish a connection"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   700
    fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   701
    data1 = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   702
    fo.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   703
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   704
    i = 20
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   705
    print "  waiting %i seconds for the server to close the connection" % i
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   706
    while i > 0:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   707
        sys.stdout.write('\r  %2i' % i)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   708
        sys.stdout.flush()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   709
        time.sleep(1)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   710
        i -= 1
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   711
    sys.stderr.write('\r')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   712
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   713
    print "  fetching the file a second time"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   714
    fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   715
    data2 = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   716
    fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   717
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   718
    if data1 == data2:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   719
        print '  data are identical'
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   720
    else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   721
        print '  ERROR: DATA DIFFER'
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   722
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   723
    DEBUG = dbbackup
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   724
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   725
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   726
def test(url, N=10):
17424
e7cfe3587ea4 fix trivial spelling errors
Mads Kiilerich <mads@kiilerich.com>
parents: 17251
diff changeset
   727
    print "checking error handler (do this on a non-200)"
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   728
    try: error_handler(url)
7875
553aa0cbeab6 cleanup: drop unused assignments
Peter Arrenbrecht <peter.arrenbrecht@gmail.com>
parents: 7874
diff changeset
   729
    except IOError:
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   730
        print "exiting - exception will prevent further tests"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   731
        sys.exit()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   732
    print
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   733
    print "performing continuity test (making sure stuff isn't corrupted)"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   734
    continuity(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   735
    print
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   736
    print "performing speed comparison"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   737
    comp(N, url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   738
    print
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   739
    print "performing dropped-connection check"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   740
    test_timeout(url)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
   741
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   742
if __name__ == '__main__':
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   743
    import time
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   744
    import sys
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   745
    try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   746
        N = int(sys.argv[1])
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   747
        url = sys.argv[2]
16688
cfb6682961b8 cleanup: replace naked excepts with more specific ones
Brodie Rao <brodie@sf.io>
parents: 16687
diff changeset
   748
    except (IndexError, ValueError):
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   749
        print "%s <integer> <url>" % sys.argv[0]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   750
    else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
   751
        test(url, N)