================

python-zstandard

================

This project provides Python bindings for interfacing with the

`Zstandard <http://www.zstd.net>`_ compression library. A C extension

and CFFI interface are provided.

The primary goal of the project is to provide a rich interface to the

underlying C API through a Pythonic interface while not sacrificing

performance. This means exposing most of the features and flexibility

of the C API while not sacrificing usability or safety that Python provides.

The canonical home for this project is

https://github.com/indygreg/python-zstandard.

|  |ci-status| |win-ci-status|

State of Project

================

The project is officially in beta state. The author is reasonably satisfied

that functionality works as advertised. **There will be some backwards

incompatible changes before 1.0, probably in the 0.9 release.** This may

involve renaming the main module from *zstd* to *zstandard* and renaming

various types and methods. Pin the package version to prevent unwanted

breakage when this change occurs!

This project is vendored and distributed with Mercurial 4.1, where it is

used in a production capacity.

There is continuous integration for Python versions 2.6, 2.7, and 3.3+

on Linux x86_x64 and Windows x86 and x86_64. The author is reasonably

confident the extension is stable and works as advertised on these

platforms.

The CFFI bindings are mostly feature complete. Where a feature is implemented

in CFFI, unit tests run against both C extension and CFFI implementation to

ensure behavior parity.

Expected Changes

----------------

The author is reasonably confident in the current state of what's

implemented on the ``ZstdCompressor`` and ``ZstdDecompressor`` types.

Those APIs likely won't change significantly. Some low-level behavior

(such as naming and types expected by arguments) may change.

There will likely be arguments added to control the input and output

buffer sizes (currently, certain operations read and write in chunk

sizes using zstd's preferred defaults).

There should be an API that accepts an object that conforms to the buffer

interface and returns an iterator over compressed or decompressed output.

There should be an API that exposes an ``io.RawIOBase`` interface to

compressor and decompressor streams, like how ``gzip.GzipFile`` from

the standard library works (issue 13).

The author is on the fence as to whether to support the extremely

low level compression and decompression APIs. It could be useful to

support compression without the framing headers. But the author doesn't

believe it a high priority at this time.

There will likely be a refactoring of the module names. Currently,

``zstd`` is a C extension and ``zstd_cffi`` is the CFFI interface.

This means that all code for the C extension must be implemented in

C. ``zstd`` may be converted to a Python module so code can be reused

between CFFI and C and so not all code in the C extension has to be C.

Requirements

============

This extension is designed to run with Python 2.6, 2.7, 3.3, 3.4, 3.5, and

3.6 on common platforms (Linux, Windows, and OS X). Only x86_64 is

currently well-tested as an architecture.

Installing

==========

This package is uploaded to PyPI at https://pypi.python.org/pypi/zstandard.

So, to install this package::

   $ pip install zstandard

Binary wheels are made available for some platforms. If you need to

install from a source distribution, all you should need is a working C

compiler and the Python development headers/libraries. On many Linux

distributions, you can install a ``python-dev`` or ``python-devel``

package to provide these dependencies.

Packages are also uploaded to Anaconda Cloud at

https://anaconda.org/indygreg/zstandard. See that URL for how to install

this package with ``conda``.

Performance

===========

Very crude and non-scientific benchmarking (most benchmarks fall in this

category because proper benchmarking is hard) show that the Python bindings

perform within 10% of the native C implementation.

The following table compares the performance of compressing and decompressing

a 1.1 GB tar file comprised of the files in a Firefox source checkout. Values

obtained with the ``zstd`` program are on the left. The remaining columns detail

performance of various compression APIs in the Python bindings.

+-------+-----------------+-----------------+-----------------+---------------+

| Level | Native          | Simple          | Stream In       | Stream Out    |

|       | Comp / Decomp   | Comp / Decomp   | Comp / Decomp   | Comp          |

+=======+=================+=================+=================+===============+

|   1   | 490 / 1338 MB/s | 458 / 1266 MB/s | 407 / 1156 MB/s |  405 MB/s     |

+-------+-----------------+-----------------+-----------------+---------------+

|   2   | 412 / 1288 MB/s | 381 / 1203 MB/s | 345 / 1128 MB/s |  349 MB/s     |

+-------+-----------------+-----------------+-----------------+---------------+

|   3   | 342 / 1312 MB/s | 319 / 1182 MB/s | 285 / 1165 MB/s |  287 MB/s     |

+-------+-----------------+-----------------+-----------------+---------------+

|  11   |  64 / 1506 MB/s |  66 / 1436 MB/s |  56 / 1342 MB/s |   57 MB/s     |

+-------+-----------------+-----------------+-----------------+---------------+

Again, these are very unscientific. But it shows that Python is capable of

compressing at several hundred MB/s and decompressing at over 1 GB/s.

Comparison to Other Python Bindings

===================================

https://pypi.python.org/pypi/zstd is an alternate Python binding to

Zstandard. At the time this was written, the latest release of that

package (1.1.2) only exposed the simple APIs for compression and decompression.

This package exposes much more of the zstd API, including streaming and

dictionary compression. This package also has CFFI support.

Bundling of Zstandard Source Code

=================================

The source repository for this project contains a vendored copy of the

Zstandard source code. This is done for a few reasons.

First, Zstandard is relatively new and not yet widely available as a system

package. Providing a copy of the source code enables the Python C extension

to be compiled without requiring the user to obtain the Zstandard source code

separately.

Second, Zstandard has both a stable *public* API and an *experimental* API.

The *experimental* API is actually quite useful (contains functionality for

training dictionaries for example), so it is something we wish to expose to

Python. However, the *experimental* API is only available via static linking.

Furthermore, the *experimental* API can change at any time. So, control over

the exact version of the Zstandard library linked against is important to

ensure known behavior.

Instructions for Building and Testing

=====================================

Once you have the source code, the extension can be built via setup.py::

   $ python setup.py build_ext

We recommend testing with ``nose``::

   $ nosetests

A Tox configuration is present to test against multiple Python versions::

   $ tox

Tests use the ``hypothesis`` Python package to perform fuzzing. If you

don't have it, those tests won't run. Since the fuzzing tests take longer

to execute than normal tests, you'll need to opt in to running them by

setting the ``ZSTD_SLOW_TESTS`` environment variable. This is set

automatically when using ``tox``.

The ``cffi`` Python package needs to be installed in order to build the CFFI

bindings. If it isn't present, the CFFI bindings won't be built.

To create a virtualenv with all development dependencies, do something

like the following::

  # Python 2

  $ virtualenv venv

  # Python 3

  $ python3 -m venv venv

  $ source venv/bin/activate

  $ pip install cffi hypothesis nose tox

API

===

The compiled C extension provides a ``zstd`` Python module. The CFFI

bindings provide a ``zstd_cffi`` module. Both provide an identical API

interface. The types, functions, and attributes exposed by these modules

are documented in the sections below.

.. note::

   The documentation in this section makes references to various zstd

   concepts and functionality. The ``Concepts`` section below explains

   these concepts in more detail.

ZstdCompressor

--------------

The ``ZstdCompressor`` class provides an interface for performing

compression operations.

Each instance is associated with parameters that control compression

behavior. These come from the following named arguments (all optional):

level

   Integer compression level. Valid values are between 1 and 22.

dict_data

   Compression dictionary to use.

   Note: When using dictionary data and ``compress()`` is called multiple

   times, the ``CompressionParameters`` derived from an integer compression

   ``level`` and the first compressed data's size will be reused for all

   subsequent operations. This may not be desirable if source data size

   varies significantly.

compression_params

   A ``CompressionParameters`` instance (overrides the ``level`` value).

write_checksum

   Whether a 4 byte checksum should be written with the compressed data.

   Defaults to False. If True, the decompressor can verify that decompressed

   data matches the original input data.

write_content_size

   Whether the size of the uncompressed data will be written into the

   header of compressed data. Defaults to False. The data will only be

   written if the compressor knows the size of the input data. This is

   likely not true for streaming compression.

write_dict_id

   Whether to write the dictionary ID into the compressed data.

   Defaults to True. The dictionary ID is only written if a dictionary

   is being used.

threads

   Enables and sets the number of threads to use for multi-threaded compression

   operations. Defaults to 0, which means to use single-threaded compression.

   Negative values will resolve to the number of logical CPUs in the system.

   Read below for more info on multi-threaded compression. This argument only

   controls thread count for operations that operate on individual pieces of

   data. APIs that spawn multiple threads for working on multiple pieces of

   data have their own ``threads`` argument.

Unless specified otherwise, assume that no two methods of ``ZstdCompressor``

instances can be called from multiple Python threads simultaneously. In other

words, assume instances are not thread safe unless stated otherwise.

Simple API

^^^^^^^^^^

``compress(data)`` compresses and returns data as a one-shot operation.::

   cctx = zstd.ZstdCompressor()

   compressed = cctx.compress(b'data to compress')

The ``data`` argument can be any object that implements the *buffer protocol*.

Unless ``compression_params`` or ``dict_data`` are passed to the

``ZstdCompressor``, each invocation of ``compress()`` will calculate the

optimal compression parameters for the configured compression ``level`` and

input data size (some parameters are fine-tuned for small input sizes).

If a compression dictionary is being used, the compression parameters

determined from the first input's size will be reused for subsequent

operations.

There is currently a deficiency in zstd's C APIs that makes it difficult

to round trip empty inputs when ``write_content_size=True``. Attempting

this will raise a ``ValueError`` unless ``allow_empty=True`` is passed

to ``compress()``.

Streaming Input API

^^^^^^^^^^^^^^^^^^^

``write_to(fh)`` (which behaves as a context manager) allows you to *stream*

data into a compressor.::

   cctx = zstd.ZstdCompressor(level=10)

   with cctx.write_to(fh) as compressor:

       compressor.write(b'chunk 0')

       compressor.write(b'chunk 1')

       ...

The argument to ``write_to()`` must have a ``write(data)`` method. As

compressed data is available, ``write()`` will be called with the compressed

data as its argument. Many common Python types implement ``write()``, including

open file handles and ``io.BytesIO``.

``write_to()`` returns an object representing a streaming compressor instance.

It **must** be used as a context manager. That object's ``write(data)`` method

is used to feed data into the compressor.

A ``flush()`` method can be called to evict whatever data remains within the

compressor's internal state into the output object. This may result in 0 or

more ``write()`` calls to the output object.

Both ``write()`` and ``flush()`` return the number of bytes written to the

object's ``write()``. In many cases, small inputs do not accumulate enough

data to cause a write and ``write()`` will return ``0``.

If the size of the data being fed to this streaming compressor is known,

you can declare it before compression begins::

   cctx = zstd.ZstdCompressor()

   with cctx.write_to(fh, size=data_len) as compressor:

       compressor.write(chunk0)

       compressor.write(chunk1)

       ...

Declaring the size of the source data allows compression parameters to

be tuned. And if ``write_content_size`` is used, it also results in the

content size being written into the frame header of the output data.

The size of chunks being ``write()`` to the destination can be specified::

    cctx = zstd.ZstdCompressor()

    with cctx.write_to(fh, write_size=32768) as compressor:

        ...

To see how much memory is being used by the streaming compressor::

    cctx = zstd.ZstdCompressor()

    with cctx.write_to(fh) as compressor:

        ...

        byte_size = compressor.memory_size()

Streaming Output API

^^^^^^^^^^^^^^^^^^^^

``read_from(reader)`` provides a mechanism to stream data out of a compressor

as an iterator of data chunks.::

   cctx = zstd.ZstdCompressor()

   for chunk in cctx.read_from(fh):

        # Do something with emitted data.

``read_from()`` accepts an object that has a ``read(size)`` method or conforms

to the buffer protocol. (``bytes`` and ``memoryview`` are 2 common types that

provide the buffer protocol.)

Uncompressed data is fetched from the source either by calling ``read(size)``

or by fetching a slice of data from the object directly (in the case where

the buffer protocol is being used). The returned iterator consists of chunks

of compressed data.

If reading from the source via ``read()``, ``read()`` will be called until

it raises or returns an empty bytes (``b''``). It is perfectly valid for

the source to deliver fewer bytes than were what requested by ``read(size)``.

Like ``write_to()``, ``read_from()`` also accepts a ``size`` argument

declaring the size of the input stream::

    cctx = zstd.ZstdCompressor()

    for chunk in cctx.read_from(fh, size=some_int):

        pass

You can also control the size that data is ``read()`` from the source and

the ideal size of output chunks::

    cctx = zstd.ZstdCompressor()

    for chunk in cctx.read_from(fh, read_size=16384, write_size=8192):

        pass

Unlike ``write_to()``, ``read_from()`` does not give direct control over the

sizes of chunks fed into the compressor. Instead, chunk sizes will be whatever

the object being read from delivers. These will often be of a uniform size.

Stream Copying API

^^^^^^^^^^^^^^^^^^

``copy_stream(ifh, ofh)`` can be used to copy data between 2 streams while

compressing it.::

   cctx = zstd.ZstdCompressor()

   cctx.copy_stream(ifh, ofh)

For example, say you wish to compress a file::

   cctx = zstd.ZstdCompressor()

   with open(input_path, 'rb') as ifh, open(output_path, 'wb') as ofh:

       cctx.copy_stream(ifh, ofh)

It is also possible to declare the size of the source stream::

   cctx = zstd.ZstdCompressor()

   cctx.copy_stream(ifh, ofh, size=len_of_input)

You can also specify how large the chunks that are ``read()`` and ``write()``

from and to the streams::

   cctx = zstd.ZstdCompressor()

   cctx.copy_stream(ifh, ofh, read_size=32768, write_size=16384)

The stream copier returns a 2-tuple of bytes read and written::

   cctx = zstd.ZstdCompressor()

   read_count, write_count = cctx.copy_stream(ifh, ofh)

Compressor API

^^^^^^^^^^^^^^

``compressobj()`` returns an object that exposes ``compress(data)`` and

``flush()`` methods. Each returns compressed data or an empty bytes.

The purpose of ``compressobj()`` is to provide an API-compatible interface

with ``zlib.compressobj`` and ``bz2.BZ2Compressor``. This allows callers to

swap in different compressor objects while using the same API.

``flush()`` accepts an optional argument indicating how to end the stream.

``zstd.COMPRESSOBJ_FLUSH_FINISH`` (the default) ends the compression stream.

Once this type of flush is performed, ``compress()`` and ``flush()`` can

no longer be called. This type of flush **must** be called to end the

compression context. If not called, returned data may be incomplete.

A ``zstd.COMPRESSOBJ_FLUSH_BLOCK`` argument to ``flush()`` will flush a

zstd block. Flushes of this type can be performed multiple times. The next

call to ``compress()`` will begin a new zstd block.

Here is how this API should be used::

   cctx = zstd.ZstdCompressor()

   cobj = cctx.compressobj()

   data = cobj.compress(b'raw input 0')

   data = cobj.compress(b'raw input 1')

   data = cobj.flush()

Or to flush blocks::

   cctx.zstd.ZstdCompressor()

   cobj = cctx.compressobj()

   data = cobj.compress(b'chunk in first block')

   data = cobj.flush(zstd.COMPRESSOBJ_FLUSH_BLOCK)

   data = cobj.compress(b'chunk in second block')

   data = cobj.flush()

For best performance results, keep input chunks under 256KB. This avoids

extra allocations for a large output object.

It is possible to declare the input size of the data that will be fed into

the compressor::

   cctx = zstd.ZstdCompressor()

   cobj = cctx.compressobj(size=6)

   data = cobj.compress(b'foobar')

   data = cobj.flush()

Batch Compression API

^^^^^^^^^^^^^^^^^^^^^

(Experimental. Not yet supported in CFFI bindings.)

``multi_compress_to_buffer(data, [threads=0])`` performs compression of multiple

inputs as a single operation.

Data to be compressed can be passed as a ``BufferWithSegmentsCollection``, a

``BufferWithSegments``, or a list containing byte like objects. Each element of

the container will be compressed individually using the configured parameters

on the ``ZstdCompressor`` instance.

The ``threads`` argument controls how many threads to use for compression. The

default is ``0`` which means to use a single thread. Negative values use the

number of logical CPUs in the machine.

The function returns a ``BufferWithSegmentsCollection``. This type represents

N discrete memory allocations, eaching holding 1 or more compressed frames.

Output data is written to shared memory buffers. This means that unlike

regular Python objects, a reference to *any* object within the collection

keeps the shared buffer and therefore memory backing it alive. This can have

undesirable effects on process memory usage.

The API and behavior of this function is experimental and will likely change.

Known deficiencies include:

* If asked to use multiple threads, it will always spawn that many threads,

  even if the input is too small to use them. It should automatically lower

  the thread count when the extra threads would just add overhead.

* The buffer allocation strategy is fixed. There is room to make it dynamic,

  perhaps even to allow one output buffer per input, facilitating a variation

  of the API to return a list without the adverse effects of shared memory

  buffers.

ZstdDecompressor

----------------

The ``ZstdDecompressor`` class provides an interface for performing

decompression.

Each instance is associated with parameters that control decompression. These

come from the following named arguments (all optional):

dict_data

   Compression dictionary to use.

The interface of this class is very similar to ``ZstdCompressor`` (by design).

Unless specified otherwise, assume that no two methods of ``ZstdDecompressor``