SparkUI

SparkUI represents the web UI for a Spark application and Spark History Server. It is created and bound when SparkContext is created (with spark.ui.enabled enabled).

Note
 The only difference between SparkUI for a Spark application and Spark History Server is that…​FIXME

When started, SparkUI binds to appUIAddress address that you can control using SPARK_PUBLIC_DNS environment variable or spark.driver.host Spark property.

Tip

Enable INFO logging level for org.apache.spark.ui.SparkUI logger to see what happens inside.

Add the following line to conf/log4j.properties:

log4j.logger.org.apache.spark.ui.SparkUI=INFO

Refer to Logging.

Creating SparkUI Instance

class SparkUI (
  val sc: Option[SparkContext],
  val conf: SparkConf,
  securityManager: SecurityManager,
  val environmentListener: EnvironmentListener,
  val storageStatusListener: StorageStatusListener,
  val executorsListener: ExecutorsListener,
  val jobProgressListener: JobProgressListener,
  val storageListener: StorageListener,
  val operationGraphListener: RDDOperationGraphListener,
  var appName: String,
  val basePath: String,
  val startTime: Long)
extends WebUI(securityManager,
  securityManager.getSSLOptions("ui"), SparkUI.getUIPort(conf),
  conf, basePath, "SparkUI")

When executed, SparkUI creates a StagesTab and initializes the tabs and handlers in web UI.

Note
 SparkUI is created when SparkContext is created (with spark.ui.enabled enabled). SparkUI gets the references to the owning SparkContext and the other properties, i.e. SparkConf, LiveListenerBus Event Bus, JobProgressListener, SecurityManager, appName, and startTime.

Attaching Tabs and Context Handlers — initialize Method

initialize(): Unit

initialize attaches the tabs of the following pages:

  1. JobsTab

  2. StagesTab

  3. StorageTab

  4. EnvironmentTab

  5. ExecutorsTab

initialize also attaches ServletContextHandler handlers:

  1. /static to serve static files from org/apache/spark/ui/static directory (on CLASSPATH).

  2. Redirecting / to /jobs/ (so Jobs tab is the first tab when you open web UI).

  3. Serving /api context path (with org.apache.spark.status.api.v1 provider package) using ApiRootResource.

  4. Redirecting /stages/stage/kill to /stages/

Note
 initialize is a part of the WebUI Contract and is executed when SparkUI is created.

Stopping SparkUI — stop Method

stop(): Unit

stop stops the HTTP server and prints the following INFO message to the logs:

INFO SparkUI: Stopped Spark web UI at [appUIAddress]
Note
 appUIAddress in the above INFO message is the result of appUIAddress method.

appUIAddress Method

appUIAddress: String

appUIAddress returns the entire URL of a Spark application’s web UI, including http:// scheme.

Internally, appUIAddress uses appUIHostPort.

getSparkUser Method

getSparkUser: String

getSparkUser returns the name of the user a Spark application runs as.

Internally, getSparkUser requests user.name System property from EnvironmentListener Spark listener.

Note
 getSparkUser is only used to display the user name in Spark Jobs page

createLiveUI Method

createLiveUI(
  sc: SparkContext,
  conf: SparkConf,
  listenerBus: SparkListenerBus,
  jobProgressListener: JobProgressListener,
  securityManager: SecurityManager,
  appName: String,
  startTime: Long): SparkUI

createLiveUI creates a SparkUI for a live running Spark application.

Internally, createLiveUI simply forwards the call to create.

Note
 createLiveUI is called when SparkContext is created (and spark.ui.enabled is enabled).

createHistoryUI Method

Caution
 FIXME

create Factory Method

create(
  sc: Option[SparkContext],
  conf: SparkConf,
  listenerBus: SparkListenerBus,
  securityManager: SecurityManager,
  appName: String,
  basePath: String = "",
  jobProgressListener: Option[JobProgressListener] = None,
  startTime: Long): SparkUI

create is a factory helper method to create a SparkUI. It is responsible for registering the SparkListeners for SparkUI.

Note
 create creates a web UI for a running Spark application and Spark History Server.

Internally, create registers a EnvironmentListener, StorageStatusListener, ExecutorsListener, StorageListener, and RDDOperationGraphListener with the input listenerBus. Once the listeners are registered, create creates an instance of SparkUI.

appUIHostPort Method

appUIHostPort: String

appUIHostPort returns the Spark application’s web UI which is the public hostname and port, excluding the scheme.

Note
 appUIAddress uses appUIHostPort and adds http:// scheme.

getAppName Method

getAppName: String

getAppName returns the name of the Spark application (of a SparkUI instance).

Note
 getAppName is used when SparkUITab is requested the application’s name.

SparkUITab — Custom WebUITab

SparkUITab is a private[spark] custom WebUITab that defines one method only, i.e. appName.

appName: String

appName returns the application’s name.

results matching ""

    No results matching ""