================

python-zstandard

================

This project provides Python bindings for interfacing with the

`Zstandard <http://www.zstd.net>`_ compression library. A C extension

and CFFI interface are provided.

The primary goal of the project is to provide a rich interface to the

underlying C API through a Pythonic interface while not sacrificing

performance. This means exposing most of the features and flexibility

of the C API while not sacrificing usability or safety that Python provides.

The canonical home for this project lives in a Mercurial repository run by

the author. For convenience, that repository is frequently synchronized to

https://github.com/indygreg/python-zstandard.

|  |ci-status|

Requirements

============

This extension is designed to run with Python 2.7, 3.5, 3.6, 3.7, and 3.8

on common platforms (Linux, Windows, and OS X). On PyPy (both PyPy2 and PyPy3) we support version 6.0.0 and above. 

x86 and x86_64 are well-tested on Windows. Only x86_64 is well-tested on Linux and macOS.

Installing

==========

This package is uploaded to PyPI at https://pypi.python.org/pypi/zstandard.

So, to install this package::

   $ pip install zstandard

Binary wheels are made available for some platforms. If you need to

install from a source distribution, all you should need is a working C

compiler and the Python development headers/libraries. On many Linux

distributions, you can install a ``python-dev`` or ``python-devel``

package to provide these dependencies.

Packages are also uploaded to Anaconda Cloud at

https://anaconda.org/indygreg/zstandard. See that URL for how to install

this package with ``conda``.

Performance

===========

zstandard is a highly tunable compression algorithm. In its default settings

(compression level 3), it will be faster at compression and decompression and

will have better compression ratios than zlib on most data sets. When tuned

for speed, it approaches lz4's speed and ratios. When tuned for compression

ratio, it approaches lzma ratios and compression speed, but decompression

speed is much faster. See the official zstandard documentation for more.

zstandard and this library support multi-threaded compression. There is a

mechanism to compress large inputs using multiple threads.

The performance of this library is usually very similar to what the zstandard

C API can deliver. Overhead in this library is due to general Python overhead

and can't easily be avoided by *any* zstandard Python binding. This library

exposes multiple APIs for performing compression and decompression so callers

can pick an API suitable for their need. Contrast with the compression

modules in Python's standard library (like ``zlib``), which only offer limited

mechanisms for performing operations. The API flexibility means consumers can

choose to use APIs that facilitate zero copying or minimize Python object

creation and garbage collection overhead.

This library is capable of single-threaded throughputs well over 1 GB/s. For

exact numbers, measure yourself. The source code repository has a ``bench.py``

script that can be used to measure things.

API

===

To interface with Zstandard, simply import the ``zstandard`` module::

   import zstandard

It is a popular convention to alias the module as a different name for

brevity::

   import zstandard as zstd

This module attempts to import and use either the C extension or CFFI

implementation. On Python platforms known to support C extensions (like

CPython), it raises an ImportError if the C extension cannot be imported.

On Python platforms known to not support C extensions (like PyPy), it only

attempts to import the CFFI implementation and raises ImportError if that

can't be done. On other platforms, it first tries to import the C extension

then falls back to CFFI if that fails and raises ImportError if CFFI fails.

To change the module import behavior, a ``PYTHON_ZSTANDARD_IMPORT_POLICY``

environment variable can be set. The following values are accepted:

default

   The behavior described above.

cffi_fallback

   Always try to import the C extension then fall back to CFFI if that

   fails.

cext

   Only attempt to import the C extension.

cffi

   Only attempt to import the CFFI implementation.

In addition, the ``zstandard`` module exports a ``backend`` attribute

containing the string name of the backend being used. It will be one

of ``cext`` or ``cffi`` (for *C extension* and *cffi*, respectively).

The types, functions, and attributes exposed by the ``zstandard`` module

are documented in the sections below.

.. note::

   The documentation in this section makes references to various zstd

   concepts and functionality. The source repository contains a

   ``docs/concepts.rst`` file explaining these in more detail.

ZstdCompressor

--------------

The ``ZstdCompressor`` class provides an interface for performing

compression operations. Each instance is essentially a wrapper around a

``ZSTD_CCtx`` from the C API.

Each instance is associated with parameters that control compression

behavior. These come from the following named arguments (all optional):

level

   Integer compression level. Valid values are between 1 and 22.

dict_data

   Compression dictionary to use.

   Note: When using dictionary data and ``compress()`` is called multiple

   times, the ``ZstdCompressionParameters`` derived from an integer

   compression ``level`` and the first compressed data's size will be reused

   for all subsequent operations. This may not be desirable if source data

   size varies significantly.

compression_params

   A ``ZstdCompressionParameters`` instance defining compression settings.

write_checksum

   Whether a 4 byte checksum should be written with the compressed data.

   Defaults to False. If True, the decompressor can verify that decompressed

   data matches the original input data.

write_content_size

   Whether the size of the uncompressed data will be written into the

   header of compressed data. Defaults to True. The data will only be

   written if the compressor knows the size of the input data. This is

   often not true for streaming compression.

write_dict_id

   Whether to write the dictionary ID into the compressed data.

   Defaults to True. The dictionary ID is only written if a dictionary

   is being used.

threads

   Enables and sets the number of threads to use for multi-threaded compression

   operations. Defaults to 0, which means to use single-threaded compression.

   Negative values will resolve to the number of logical CPUs in the system.

   Read below for more info on multi-threaded compression. This argument only

   controls thread count for operations that operate on individual pieces of

   data. APIs that spawn multiple threads for working on multiple pieces of

   data have their own ``threads`` argument.

``compression_params`` is mutually exclusive with ``level``, ``write_checksum``,

``write_content_size``, ``write_dict_id``, and ``threads``.

Unless specified otherwise, assume that no two methods of ``ZstdCompressor``

instances can be called from multiple Python threads simultaneously. In other

words, assume instances are not thread safe unless stated otherwise.

Utility Methods

^^^^^^^^^^^^^^^

``frame_progression()`` returns a 3-tuple containing the number of bytes

ingested, consumed, and produced by the current compression operation.

``memory_size()`` obtains the memory utilization of the underlying zstd

compression context, in bytes.::

    cctx = zstd.ZstdCompressor()

    memory = cctx.memory_size()

Simple API

^^^^^^^^^^

``compress(data)`` compresses and returns data as a one-shot operation.::

   cctx = zstd.ZstdCompressor()

   compressed = cctx.compress(b'data to compress')

The ``data`` argument can be any object that implements the *buffer protocol*.

Stream Reader API

^^^^^^^^^^^^^^^^^

``stream_reader(source)`` can be used to obtain an object conforming to the

``io.RawIOBase`` interface for reading compressed output as a stream::

   with open(path, 'rb') as fh:

       cctx = zstd.ZstdCompressor()

       reader = cctx.stream_reader(fh)

       while True:

           chunk = reader.read(16384)

           if not chunk:

               break

           # Do something with compressed chunk.

Instances can also be used as context managers::

   with open(path, 'rb') as fh:

       with cctx.stream_reader(fh) as reader:

           while True:

               chunk = reader.read(16384)

               if not chunk:

                   break

               # Do something with compressed chunk.

When the context manager exits or ``close()`` is called, the stream is closed,

underlying resources are released, and future operations against the compression

stream will fail.

The ``source`` argument to ``stream_reader()`` can be any object with a

``read(size)`` method or any object implementing the *buffer protocol*.

``stream_reader()`` accepts a ``size`` argument specifying how large the input

stream is. This is used to adjust compression parameters so they are

tailored to the source size.::

   with open(path, 'rb') as fh:

       cctx = zstd.ZstdCompressor()

       with cctx.stream_reader(fh, size=os.stat(path).st_size) as reader:

           ...

If the ``source`` is a stream, you can specify how large ``read()`` requests

to that stream should be via the ``read_size`` argument. It defaults to

``zstandard.COMPRESSION_RECOMMENDED_INPUT_SIZE``.::

   with open(path, 'rb') as fh:

       cctx = zstd.ZstdCompressor()

       # Will perform fh.read(8192) when obtaining data to feed into the

       # compressor.

       with cctx.stream_reader(fh, read_size=8192) as reader:

           ...

The stream returned by ``stream_reader()`` is neither writable nor seekable

(even if the underlying source is seekable). ``readline()`` and

``readlines()`` are not implemented because they don't make sense for

compressed data. ``tell()`` returns the number of compressed bytes

emitted so far.

Streaming Input API

^^^^^^^^^^^^^^^^^^^

``stream_writer(fh)`` allows you to *stream* data into a compressor.

Returned instances implement the ``io.RawIOBase`` interface. Only methods

that involve writing will do useful things.

The argument to ``stream_writer()`` must have a ``write(data)`` method. As

compressed data is available, ``write()`` will be called with the compressed

data as its argument. Many common Python types implement ``write()``, including

open file handles and ``io.BytesIO``.

The ``write(data)`` method is used to feed data into the compressor.

The ``flush([flush_mode=FLUSH_BLOCK])`` method can be called to evict whatever

data remains within the compressor's internal state into the output object. This

may result in 0 or more ``write()`` calls to the output object. This method

accepts an optional ``flush_mode`` argument to control the flushing behavior.

Its value can be any of the ``FLUSH_*`` constants.

Both ``write()`` and ``flush()`` return the number of bytes written to the

object's ``write()``. In many cases, small inputs do not accumulate enough

data to cause a write and ``write()`` will return ``0``.

Calling ``close()`` will mark the stream as closed and subsequent I/O

operations will raise ``ValueError`` (per the documented behavior of

``io.RawIOBase``). ``close()`` will also call ``close()`` on the underlying

stream if such a method exists.

Typically usage is as follows::

   cctx = zstd.ZstdCompressor(level=10)

   compressor = cctx.stream_writer(fh)

   compressor.write(b'chunk 0\n')

   compressor.write(b'chunk 1\n')

   compressor.flush()

   # Receiver will be able to decode ``chunk 0\nchunk 1\n`` at this point.

   # Receiver is also expecting more data in the zstd *frame*.

   compressor.write(b'chunk 2\n')

   compressor.flush(zstd.FLUSH_FRAME)

   # Receiver will be able to decode ``chunk 0\nchunk 1\nchunk 2``.

   # Receiver is expecting no more data, as the zstd frame is closed.

   # Any future calls to ``write()`` at this point will construct a new

   # zstd frame.

Instances can be used as context managers. Exiting the context manager is

the equivalent of calling ``close()``, which is equivalent to calling

``flush(zstd.FLUSH_FRAME)``::

   cctx = zstd.ZstdCompressor(level=10)

   with cctx.stream_writer(fh) as compressor:

       compressor.write(b'chunk 0')

       compressor.write(b'chunk 1')

       ...

.. important::

   If ``flush(FLUSH_FRAME)`` is not called, emitted data doesn't constitute

   a full zstd *frame* and consumers of this data may complain about malformed

   input. It is recommended to use instances as a context manager to ensure

   *frames* are properly finished.

If the size of the data being fed to this streaming compressor is known,

you can declare it before compression begins::

   cctx = zstd.ZstdCompressor()

   with cctx.stream_writer(fh, size=data_len) as compressor:

       compressor.write(chunk0)

       compressor.write(chunk1)

       ...

Declaring the size of the source data allows compression parameters to

be tuned. And if ``write_content_size`` is used, it also results in the

content size being written into the frame header of the output data.

The size of chunks being ``write()`` to the destination can be specified::

    cctx = zstd.ZstdCompressor()

    with cctx.stream_writer(fh, write_size=32768) as compressor:

        ...

To see how much memory is being used by the streaming compressor::

    cctx = zstd.ZstdCompressor()

    with cctx.stream_writer(fh) as compressor:

        ...

        byte_size = compressor.memory_size()

Thte total number of bytes written so far are exposed via ``tell()``::

    cctx = zstd.ZstdCompressor()

    with cctx.stream_writer(fh) as compressor:

        ...

        total_written = compressor.tell()

``stream_writer()`` accepts a ``write_return_read`` boolean argument to control

the return value of ``write()``. When ``False`` (the default), ``write()`` returns

the number of bytes that were ``write()``en to the underlying object. When

``True``, ``write()`` returns the number of bytes read from the input that

were subsequently written to the compressor. ``True`` is the *proper* behavior

for ``write()`` as specified by the ``io.RawIOBase`` interface and will become

the default value in a future release.

Streaming Output API

^^^^^^^^^^^^^^^^^^^^

``read_to_iter(reader)`` provides a mechanism to stream data out of a

compressor as an iterator of data chunks.::

   cctx = zstd.ZstdCompressor()

   for chunk in cctx.read_to_iter(fh):

        # Do something with emitted data.

``read_to_iter()`` accepts an object that has a ``read(size)`` method or

conforms to the buffer protocol.

Uncompressed data is fetched from the source either by calling ``read(size)``

or by fetching a slice of data from the object directly (in the case where

the buffer protocol is being used). The returned iterator consists of chunks

of compressed data.

If reading from the source via ``read()``, ``read()`` will be called until

it raises or returns an empty bytes (``b''``). It is perfectly valid for

the source to deliver fewer bytes than were what requested by ``read(size)``.

Like ``stream_writer()``, ``read_to_iter()`` also accepts a ``size`` argument

declaring the size of the input stream::

    cctx = zstd.ZstdCompressor()

    for chunk in cctx.read_to_iter(fh, size=some_int):

        pass

You can also control the size that data is ``read()`` from the source and

the ideal size of output chunks::

    cctx = zstd.ZstdCompressor()

    for chunk in cctx.read_to_iter(fh, read_size=16384, write_size=8192):

        pass

Unlike ``stream_writer()``, ``read_to_iter()`` does not give direct control

over the sizes of chunks fed into the compressor. Instead, chunk sizes will

be whatever the object being read from delivers. These will often be of a

uniform size.

Stream Copying API

^^^^^^^^^^^^^^^^^^

``copy_stream(ifh, ofh)`` can be used to copy data between 2 streams while

compressing it.::

   cctx = zstd.ZstdCompressor()

   cctx.copy_stream(ifh, ofh)

For example, say you wish to compress a file::

   cctx = zstd.ZstdCompressor()

   with open(input_path, 'rb') as ifh, open(output_path, 'wb') as ofh:

       cctx.copy_stream(ifh, ofh)

It is also possible to declare the size of the source stream::

   cctx = zstd.ZstdCompressor()

   cctx.copy_stream(ifh, ofh, size=len_of_input)

You can also specify how large the chunks that are ``read()`` and ``write()``

from and to the streams::

   cctx = zstd.ZstdCompressor()

   cctx.copy_stream(ifh, ofh, read_size=32768, write_size=16384)

The stream copier returns a 2-tuple of bytes read and written::

   cctx = zstd.ZstdCompressor()

   read_count, write_count = cctx.copy_stream(ifh, ofh)

Compressor API

^^^^^^^^^^^^^^

``compressobj()`` returns an object that exposes ``compress(data)`` and

``flush()`` methods. Each returns compressed data or an empty bytes.

The purpose of ``compressobj()`` is to provide an API-compatible interface

with ``zlib.compressobj``, ``bz2.BZ2Compressor``, etc. This allows callers to

swap in different compressor objects while using the same API.

``flush()`` accepts an optional argument indicating how to end the stream.

``zstd.COMPRESSOBJ_FLUSH_FINISH`` (the default) ends the compression stream.

Once this type of flush is performed, ``compress()`` and ``flush()`` can

no longer be called. This type of flush **must** be called to end the

compression context. If not called, returned data may be incomplete.

A ``zstd.COMPRESSOBJ_FLUSH_BLOCK`` argument to ``flush()`` will flush a

zstd block. Flushes of this type can be performed multiple times. The next

call to ``compress()`` will begin a new zstd block.

Here is how this API should be used::

   cctx = zstd.ZstdCompressor()

   cobj = cctx.compressobj()

   data = cobj.compress(b'raw input 0')

   data = cobj.compress(b'raw input 1')

   data = cobj.flush()

Or to flush blocks::

   cctx.zstd.ZstdCompressor()

   cobj = cctx.compressobj()

   data = cobj.compress(b'chunk in first block')

   data = cobj.flush(zstd.COMPRESSOBJ_FLUSH_BLOCK)

   data = cobj.compress(b'chunk in second block')

   data = cobj.flush()

For best performance results, keep input chunks under 256KB. This avoids

extra allocations for a large output object.

It is possible to declare the input size of the data that will be fed into

the compressor::

   cctx = zstd.ZstdCompressor()

   cobj = cctx.compressobj(size=6)

   data = cobj.compress(b'foobar')

   data = cobj.flush()

Chunker API

^^^^^^^^^^^

``chunker(size=None, chunk_size=COMPRESSION_RECOMMENDED_OUTPUT_SIZE)`` returns

an object that can be used to iteratively feed chunks of data into a compressor

and produce output chunks of a uniform size.