tests/test-cbor.py
author Gregory Szorc <gregory.szorc@gmail.com>
Sat, 14 Apr 2018 16:36:15 -0700
changeset 37711 65a23cc8e75b
child 37899 2b3b6187c316
permissions -rw-r--r--
cborutil: implement support for streaming encoding, bytestring decoding The vendored cbor2 package is... a bit disappointing. On the encoding side, it insists that you pass it something with a write() to send data to. That means if you want to emit data to a generator, you have to construct an e.g. io.BytesIO(), write() to it, then get the data back out. There can be non-trivial overhead involved. The encoder also doesn't support indefinite types - bytestrings, arrays, and maps that don't have a known length. Again, this is really unfortunate because it requires you to buffer the entire source and destination in memory to encode large things. On the decoding side, it supports reading indefinite length types. But it buffers them completely before returning. More sadness. This commit implements "streaming" encoders for various CBOR types. Encoding emits a generator of hunks. So you can efficiently stream encoded data elsewhere. It also implements support for emitting indefinite length bytestrings, arrays, and maps. On the decoding side, we only implement support for decoding an indefinite length bytestring from a file object. It will emit a generator of raw chunks from the source. I didn't want to reinvent so many wheels. But profiling the wire protocol revealed that the overhead of constructing io.BytesIO() instances to temporarily hold results has a non-trivial overhead. We're talking >15% of execution time for operations like "transfer the fulltexts of all files in a revision." So I can justify this effort. Fortunately, CBOR is a relatively straightforward format. And we have a reference implementation in the repo we can test against. Differential Revision: https://phab.mercurial-scm.org/D3303
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
37711
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     1
from __future__ import absolute_import
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     2
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     3
import io
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     4
import unittest
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     5
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     6
from mercurial.thirdparty import (
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     7
    cbor,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     8
)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
     9
from mercurial.utils import (
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    10
    cborutil,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    11
)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    12
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    13
def loadit(it):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    14
    return cbor.loads(b''.join(it))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    15
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    16
class BytestringTests(unittest.TestCase):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    17
    def testsimple(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    18
        self.assertEqual(
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    19
            list(cborutil.streamencode(b'foobar')),
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    20
            [b'\x46', b'foobar'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    21
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    22
        self.assertEqual(
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    23
            loadit(cborutil.streamencode(b'foobar')),
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    24
            b'foobar')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    25
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    26
    def testlong(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    27
        source = b'x' * 1048576
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    28
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    29
        self.assertEqual(loadit(cborutil.streamencode(source)), source)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    30
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    31
    def testfromiter(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    32
        # This is the example from RFC 7049 Section 2.2.2.
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    33
        source = [b'\xaa\xbb\xcc\xdd', b'\xee\xff\x99']
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    34
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    35
        self.assertEqual(
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    36
            list(cborutil.streamencodebytestringfromiter(source)),
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    37
            [
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    38
                b'\x5f',
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    39
                b'\x44',
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    40
                b'\xaa\xbb\xcc\xdd',
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    41
                b'\x43',
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    42
                b'\xee\xff\x99',
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    43
                b'\xff',
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    44
            ])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    45
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    46
        self.assertEqual(
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    47
            loadit(cborutil.streamencodebytestringfromiter(source)),
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    48
            b''.join(source))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    49
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    50
    def testfromiterlarge(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    51
        source = [b'a' * 16, b'b' * 128, b'c' * 1024, b'd' * 1048576]
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    52
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    53
        self.assertEqual(
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    54
            loadit(cborutil.streamencodebytestringfromiter(source)),
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    55
            b''.join(source))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    56
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    57
    def testindefinite(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    58
        source = b'\x00\x01\x02\x03' + b'\xff' * 16384
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    59
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    60
        it = cborutil.streamencodeindefinitebytestring(source, chunksize=2)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    61
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    62
        self.assertEqual(next(it), b'\x5f')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    63
        self.assertEqual(next(it), b'\x42')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    64
        self.assertEqual(next(it), b'\x00\x01')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    65
        self.assertEqual(next(it), b'\x42')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    66
        self.assertEqual(next(it), b'\x02\x03')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    67
        self.assertEqual(next(it), b'\x42')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    68
        self.assertEqual(next(it), b'\xff\xff')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    69
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    70
        dest = b''.join(cborutil.streamencodeindefinitebytestring(
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    71
            source, chunksize=42))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    72
        self.assertEqual(cbor.loads(dest), b''.join(source))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    73
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    74
    def testreadtoiter(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    75
        source = io.BytesIO(b'\x5f\x44\xaa\xbb\xcc\xdd\x43\xee\xff\x99\xff')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    76
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    77
        it = cborutil.readindefinitebytestringtoiter(source)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    78
        self.assertEqual(next(it), b'\xaa\xbb\xcc\xdd')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    79
        self.assertEqual(next(it), b'\xee\xff\x99')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    80
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    81
        with self.assertRaises(StopIteration):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    82
            next(it)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    83
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    84
class IntTests(unittest.TestCase):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    85
    def testsmall(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    86
        self.assertEqual(list(cborutil.streamencode(0)), [b'\x00'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    87
        self.assertEqual(list(cborutil.streamencode(1)), [b'\x01'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    88
        self.assertEqual(list(cborutil.streamencode(2)), [b'\x02'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    89
        self.assertEqual(list(cborutil.streamencode(3)), [b'\x03'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    90
        self.assertEqual(list(cborutil.streamencode(4)), [b'\x04'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    91
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    92
    def testnegativesmall(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    93
        self.assertEqual(list(cborutil.streamencode(-1)), [b'\x20'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    94
        self.assertEqual(list(cborutil.streamencode(-2)), [b'\x21'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    95
        self.assertEqual(list(cborutil.streamencode(-3)), [b'\x22'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    96
        self.assertEqual(list(cborutil.streamencode(-4)), [b'\x23'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    97
        self.assertEqual(list(cborutil.streamencode(-5)), [b'\x24'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    98
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
    99
    def testrange(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   100
        for i in range(-70000, 70000, 10):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   101
            self.assertEqual(
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   102
                b''.join(cborutil.streamencode(i)),
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   103
                cbor.dumps(i))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   104
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   105
class ArrayTests(unittest.TestCase):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   106
    def testempty(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   107
        self.assertEqual(list(cborutil.streamencode([])), [b'\x80'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   108
        self.assertEqual(loadit(cborutil.streamencode([])), [])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   109
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   110
    def testbasic(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   111
        source = [b'foo', b'bar', 1, -10]
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   112
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   113
        self.assertEqual(list(cborutil.streamencode(source)), [
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   114
            b'\x84', b'\x43', b'foo', b'\x43', b'bar', b'\x01', b'\x29'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   115
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   116
    def testemptyfromiter(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   117
        self.assertEqual(b''.join(cborutil.streamencodearrayfromiter([])),
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   118
                         b'\x9f\xff')
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   119
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   120
    def testfromiter1(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   121
        source = [b'foo']
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   122
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   123
        self.assertEqual(list(cborutil.streamencodearrayfromiter(source)), [
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   124
            b'\x9f',
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   125
            b'\x43', b'foo',
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   126
            b'\xff',
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   127
        ])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   128
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   129
        dest = b''.join(cborutil.streamencodearrayfromiter(source))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   130
        self.assertEqual(cbor.loads(dest), source)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   131
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   132
    def testtuple(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   133
        source = (b'foo', None, 42)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   134
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   135
        self.assertEqual(cbor.loads(b''.join(cborutil.streamencode(source))),
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   136
                         list(source))
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   137
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   138
class SetTests(unittest.TestCase):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   139
    def testempty(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   140
        self.assertEqual(list(cborutil.streamencode(set())), [
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   141
            b'\xd9\x01\x02',
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   142
            b'\x80',
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   143
        ])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   144
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   145
    def testset(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   146
        source = {b'foo', None, 42}
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   147
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   148
        self.assertEqual(cbor.loads(b''.join(cborutil.streamencode(source))),
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   149
                         source)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   150
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   151
class BoolTests(unittest.TestCase):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   152
    def testbasic(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   153
        self.assertEqual(list(cborutil.streamencode(True)),  [b'\xf5'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   154
        self.assertEqual(list(cborutil.streamencode(False)), [b'\xf4'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   155
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   156
        self.assertIs(loadit(cborutil.streamencode(True)), True)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   157
        self.assertIs(loadit(cborutil.streamencode(False)), False)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   158
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   159
class NoneTests(unittest.TestCase):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   160
    def testbasic(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   161
        self.assertEqual(list(cborutil.streamencode(None)), [b'\xf6'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   162
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   163
        self.assertIs(loadit(cborutil.streamencode(None)), None)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   164
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   165
class MapTests(unittest.TestCase):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   166
    def testempty(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   167
        self.assertEqual(list(cborutil.streamencode({})), [b'\xa0'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   168
        self.assertEqual(loadit(cborutil.streamencode({})), {})
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   169
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   170
    def testemptyindefinite(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   171
        self.assertEqual(list(cborutil.streamencodemapfromiter([])), [
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   172
            b'\xbf', b'\xff'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   173
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   174
        self.assertEqual(loadit(cborutil.streamencodemapfromiter([])), {})
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   175
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   176
    def testone(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   177
        source = {b'foo': b'bar'}
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   178
        self.assertEqual(list(cborutil.streamencode(source)), [
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   179
            b'\xa1', b'\x43', b'foo', b'\x43', b'bar'])
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   180
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   181
        self.assertEqual(loadit(cborutil.streamencode(source)), source)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   182
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   183
    def testmultiple(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   184
        source = {
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   185
            b'foo': b'bar',
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   186
            b'baz': b'value1',
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   187
        }
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   188
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   189
        self.assertEqual(loadit(cborutil.streamencode(source)), source)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   190
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   191
        self.assertEqual(
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   192
            loadit(cborutil.streamencodemapfromiter(source.items())),
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   193
            source)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   194
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   195
    def testcomplex(self):
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   196
        source = {
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   197
            b'key': 1,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   198
            2: -10,
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   199
        }
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   200
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   201
        self.assertEqual(loadit(cborutil.streamencode(source)),
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   202
                         source)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   203
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   204
        self.assertEqual(
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   205
            loadit(cborutil.streamencodemapfromiter(source.items())),
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   206
            source)
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   207
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   208
if __name__ == '__main__':
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   209
    import silenttestrunner
65a23cc8e75b cborutil: implement support for streaming encoding, bytestring decoding
Gregory Szorc <gregory.szorc@gmail.com>
parents:
diff changeset
   210
    silenttestrunner.main(__name__)