CompressionCodec

With spark.broadcast.compress enabled (which is the default), TorrentBroadcast uses compression for broadcast blocks.

Caution

FIXME What’s compressed?

Table 1. Built-in Compression Codecs
Codec Alias	Fully-Qualified Class Name	Notes
`lz4`	`org.apache.spark.io.LZ4CompressionCodec`	The default implementation
`lzf`	`org.apache.spark.io.LZFCompressionCodec`
`snappy`	`org.apache.spark.io.SnappyCompressionCodec`	The fallback when the default codec is not available.

An implementation of CompressionCodec trait has to offer a constructor that accepts a single argument being SparkConf. Read Creating CompressionCodec — createCodec Factory Method in this document.

You can control the default compression codec in a Spark application using spark.io.compression.codec Spark property.

createCodec(conf: SparkConf): CompressionCodec  (1)
createCodec(conf: SparkConf, codecName: String): CompressionCodec (2)

createCodec uses the internal shortCompressionCodecNames lookup table to find the input codecName (regardless of the case).

createCodec finds the constructor of the compression codec’s implementation (that accepts a single argument being SparkConf).

If a compression codec could not be found, createCodec throws a IllegalArgumentException exception:

Codec [<codecName>] is not available. Consider setting spark.io.compression.codec=snappy

getCodecName(conf: SparkConf): String

getCodecName reads spark.io.compression.codec Spark property from the input conf SparkConf or assumes lz4.

Note	`getCodecName` is used when `SparkContext` sets up event logging (for History Server) or when creating a `CompressionCodec`.

Table 2. Settings
Name	Default value	Description
`spark.io.compression.codec`	`lz4`	The compression codec to use. Used when getCodecName is called to find the current compression codec.

results matching ""