SQLConf

SQLConf is a key-value configuration store for parameters and hints used in Spark SQL. It offers methods to get, set, unset or clear their values.

You can access the current SQLConf using sparkSession.conf.

Note
 SQLConf is a private[sql] serializable class in org.apache.spark.sql.internal package.

Getting Parameters and Hints

You can get the current parameters and hints using the following family of get methods.

getConfString(key: String): String
getConf[T](entry: ConfigEntry[T], defaultValue: T): T
getConf[T](entry: ConfigEntry[T]): T
getConf[T](entry: OptionalConfigEntry[T]): Option[T]
getConfString(key: String, defaultValue: String): String
getAllConfs: immutable.Map[String, String]
getAllDefinedConfs: Seq[(String, String, String)]

Setting Parameters and Hints

You can set parameters and hints using the following family of set methods.

setConf(props: Properties): Unit
setConfString(key: String, value: String): Unit
setConf[T](entry: ConfigEntry[T], value: T): Unit

Unsetting Parameters and Hints

You can unset parameters and hints using the following family of unset methods.

unsetConf(key: String): Unit
unsetConf(entry: ConfigEntry[_]): Unit

Clearing All Parameters and Hints

clear(): Unit

You can use clear to remove all the parameters and hints in SQLConf.

Parameters and Hints

Caution
 FIXME

spark.sql.streaming.fileSink.log.deletion

spark.sql.streaming.fileSink.log.deletion (default: true) is an internal flag to control whether to delete the expired log files in file stream sink.

spark.sql.streaming.fileSink.log.compactInterval

spark.sql.streaming.fileSink.log.compactInterval

spark.sql.streaming.fileSink.log.cleanupDelay

spark.sql.streaming.fileSink.log.cleanupDelay

spark.sql.streaming.schemaInference

spark.sql.streaming.schemaInference

results matching ""

    No results matching ""