Package {zukomp}

zukomp: Portable Byte Compression with a Pluggable Codec Registry

Description

Compresses and decompresses raw vectors through a single codec-neutral interface backed by a runtime registry, so the set of available codecs is discovered rather than fixed at compile time. The 'DEFLATE' family ('gzip', 'zlib', and headerless 'DEFLATE') is built in from vendored 'miniz' sources, requiring no system compression library. Decompression enforces configurable output-size and expansion-ratio limits and reports failures through structured conditions. A registered C-callable interface lets other packages drive the codecs from C and register codecs of their own. The formats are specified in Deutsch (1996) doi:10.17487/RFC1951, Deutsch and Gailly (1996) doi:10.17487/RFC1950, and Deutsch (1996) doi:10.17487/RFC1952.

Author(s)

Maintainer: Pedro Baltazar pedrobtz@gmail.com [copyright holder]

Authors:

• Pedro Baltazar pedrobtz@gmail.com [copyright holder]

Other contributors:

• Rich Geldreich (author of the bundled miniz library) [contributor, copyright holder]

• Tenacious Software LLC (copyright holder of the bundled miniz library) [copyright holder]

• RAD Game Tools (copyright holder of the bundled miniz library) [copyright holder]

• Valve Software (copyright holder of the bundled miniz library) [copyright holder]

• Martin Raiber (contributor to and copyright holder of the bundled miniz library) [contributor, copyright holder]

• Alex Evans (author of the PNG writer in the bundled miniz library) [contributor]

• Alistair Moffat (co-author of the code-length algorithm in the bundled miniz library) [contributor]

• Jyrki Katajainen (co-author of the code-length algorithm in the bundled miniz library) [contributor]

See Also

Useful links:

• https://github.com/pedrobtz/zukomp

• https://pedrobtz.github.io/zukomp/

• Report bugs at https://github.com/pedrobtz/zukomp/issues

Is a codec implementation available?

Description

Is a codec implementation available?

Usage

komp_codec_available(codec)

Arguments

Value

TRUE if an implementation is registered, FALSE if the codec is known to zukomp but not installed. An unknown name is an error of class zukomp_unsupported_codec, since it is far more likely to be a typo than a deliberate probe.

Examples

komp_codec_available("identity")

# a codec name zukomp reserves, with no implementation released yet
komp_codec_available("zstd")

The codec capability table

Description

Every codec this build of zukomp knows the name of, whether or not an implementation is present, plus any codec registered by another package. This is the R-visible face of the registry: it is how you discover what can be compressed, at what levels, and which HTTP content-coding each codec corresponds to.

Usage

komp_codecs()

Details

A codec that is declared but not installed still gets a row, with available = FALSE and NA for everything capability-shaped. That is deliberate: "zstd exists but you need the zukomp.zstd package" is a more useful answer than pretending the codec does not exist.

Value

A data frame with one row per codec and the columns:

id

Codec name, as accepted by the codec argument elsewhere.

available

Is an implementation registered?

can_encode, can_decode

Supported directions; NA if unavailable.

level_min, level_max, level_default

Codec-native compression levels. NA when the codec has no level axis, and when it is unavailable. Levels are not comparable between codecs.

level_fast, level_best

Where the abstract levels "fast" and "best" land for this codec. These are not level_min and level_max: for the DEFLATE family level 0 is stored blocks, so "fast" is 1, and a codec whose level is an acceleration factor inverts the mapping entirely. Only the codec knows, so it declares them.

detectable

Can komp_detect() recognise this codec from its bytes? Headerless formats cannot be detected and must be named.

can_flush

Does the codec support a mid-stream flush – "put the bytes on the wire now"? NA if unavailable. A caller streaming a request body should check this before committing to a codec, rather than discovering it mid-body.

content_encoding

The HTTP content-coding token, or NA.

source

Package that registered the implementation, or NA.

References

The built-in codecs implement the formats specified in Deutsch, P. (1996) "DEFLATE Compressed Data Format Specification version 1.3", RFC 1951, doi:10.17487/RFC1951; Deutsch, P. and Gailly, J-L. (1996) "ZLIB Compressed Data Format Specification version 3.3", RFC 1950, doi:10.17487/RFC1950; and Deutsch, P. (1996) "GZIP file format specification version 4.3", RFC 1952, doi:10.17487/RFC1952.

Examples

codecs <- komp_codecs()
codecs[, c("id", "available", "content_encoding")]

# what this build can actually decompress right now
codecs$id[which(codecs$can_decode)]

Compress a raw vector

Description

Bytes in, bytes out. Character input is not accepted: encoding is a decision the caller must make explicitly, so convert with charToRaw() or iconv() first.

Usage

komp_compress(x, codec = "gzip", level = NULL)

Arguments

Value

A raw vector.

See Also

komp_decompress(), komp_codecs()

Examples

x <- charToRaw(strrep("compress me ", 100))
z <- komp_compress(x)
length(x)
length(z)
identical(komp_decompress(z, "gzip"), x)

# gzip output is deterministic: no timestamp, no filename
identical(komp_compress(x), komp_compress(x))

# abstract levels port across codecs; numeric ones do not
length(komp_compress(x, level = "fast"))
length(komp_compress(x, level = "best"))

Decompress a raw vector

Description

Decompress a raw vector

Usage

komp_decompress(
  x,
  codec = "auto",
  max_output = getOption("zukomp.max_output", 1024^3),
  max_ratio = getOption("zukomp.max_ratio", NULL)
)

Arguments

Value

A raw vector.

See Also

komp_compress(), komp_codecs()

Examples

z <- komp_compress(charToRaw(strrep("data ", 200)), "gzip")
rawToChar(komp_decompress(z, "gzip"))

# a small input that would expand a long way is stopped, not allocated
bomb <- komp_compress(raw(1e6), "gzip")
length(bomb)
try(komp_decompress(bomb, "gzip", max_output = 1024))

Identify the codec that produced a compressed stream

Description

Detection is a property of the codec registry, not a hardcoded list: a codec registered by another package becomes detectable as soon as it advertises magic bytes.

Usage

komp_detect(x)

Arguments

Details

Not every format can be detected. Raw DEFLATE, raw LZ4 blocks, raw Snappy and brotli have no header to recognise, so they return NA and must be named explicitly. This is deliberate: guessing wrong for a headerless format means silently returning the wrong bytes.

zlib is recognised by a header predicate rather than a constant, so it carries a small false-positive rate – roughly one arbitrary byte pair in a thousand looks like a valid zlib header. Codecs with real magic are always tested first.

Value

The codec's name, or NA_character_ if nothing matched.

See Also

komp_codecs(), whose detectable column says which codecs can be found this way.

Examples

komp_detect(komp_compress(charToRaw("hello"), "gzip"))
komp_detect(komp_compress(charToRaw("hello"), "zlib"))

# headerless formats are not guessed at
komp_detect(komp_compress(charToRaw("hello"), "deflate-raw"))

Build and provenance information

Description

What this build of zukomp actually contains: its version, the ABI version other packages link against, the codecs it registered, and the vendored sources compiled into it with the trim applied to them.

Usage

komp_info()

Details

Reported from the compiled library rather than read from tools/vendor/manifest.tsv, because the manifest is not installed and what matters here is what was actually built.

Value

A list with version, abi_version, codecs, vendored and build_flags. version is a character string rather than a package_version object: this is diagnostic output that mostly ends up pasted into a log line or a bug report, and paste() on a package_version needs an as.character() at every call site. Compare against utils::packageVersion() with package_version(komp_info()$version) if you need ordering.

Examples

info <- komp_info()
info$version
info$vendored
info$build_flags