StorageStatusListener Spark Listener

StorageStatusListener is a SparkListener that tracks status of BlockManagers on "nodes" in a Spark application, i.e. the driver and the executors.

Note
 StorageStatusListener is created and registered when a SparkUI is created. It is later used to create ExecutorsListener and StorageListener Spark listeners.
Table 1. StorageStatusListener Registries
Registry Description

executorIdToStorageStatus

The lookup table for StorageStatus per executor or the driver.

deadExecutorStorageStatus

A collection of StorageStatuss of removed/inactive BlockManagers.
Table 2. StorageStatusListener Event Handlers
Event Handler Description

onUnpersistRDD

Removes the RDD blocks for the unpersisted rddId from every StorageStatuss in executorIdToStorageStatus internal registry.

onBlockManagerAdded

Registers a BlockManager on an executor in executorIdToStorageStatus internal registry.

Removes any other BlockManager that may have been registered for the executor earlier in deadExecutorStorageStatus internal registry.

onBlockManagerRemoved

Removes an executor from executorIdToStorageStatus internal registry and adds the removed StorageStatus to deadExecutorStorageStatus internal registry.

Removes the oldest StorageStatus when the number of entries in deadExecutorStorageStatus is bigger than spark.ui.retainedDeadExecutors.

onBlockUpdated

Updates StorageStatus for an executor in executorIdToStorageStatus internal registry, i.e. removes a block for NONE storage level and updates otherwise.

results matching ""

    No results matching ""