log4j.logger.org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint=INFOYarnSchedulerEndpoint RPC Endpoint
YarnSchedulerEndpoint is a thread-safe RPC endpoint for communication between YarnSchedulerBackend on the driver and ApplicationMaster on YARN (inside a YARN container).
|
Caution
|
FIXME Picture it. |
It uses the reference to the remote ApplicationMaster RPC Endpoint to send messages to.
|
Tip
|
Enable Add the following line to Refer to Logging. |
RPC Messages
RequestExecutors
RequestExecutors(
requestedTotal: Int,
localityAwareTasks: Int,
hostToLocalTaskCount: Map[String, Int])
extends CoarseGrainedClusterMessageRequestExecutors is to inform ApplicationMaster about the current requirements for the total number of executors (as requestedTotal), including already pending and running executors.
Figure 1. RequestExecutors Message Flow (client deploy mode)
When a RequestExecutors arrives, YarnSchedulerEndpoint simply passes it on to ApplicationMaster (via the internal RPC endpoint reference). The result of the forward call is sent back in response.
Any issues communicating with the remote ApplicationMaster RPC endpoint are reported as ERROR messages in the logs:
ERROR Sending RequestExecutors to AM was unsuccessfulAddWebUIFilter
AddWebUIFilter(
filterName: String,
filterParams: Map[String, String],
proxyBase: String)AddWebUIFilter triggers setting spark.ui.proxyBase system property and adding the filterName filter to web UI.
AddWebUIFilter is sent by ApplicationMaster when it adds AmIpFilter to web UI.
It firstly sets spark.ui.proxyBase system property to the input proxyBase (if not empty).
If it defines a filter, i.e. the input filterName and filterParams are both not empty, you should see the following INFO message in the logs:
INFO Add WebUI Filter. [filterName], [filterParams], [proxyBase]It then sets spark.ui.filters to be the input filterName in the internal conf SparkConf attribute.
All the filterParams are also set as spark.[filterName].param.[key] and [value].
The filter is added to web UI using JettyUtils.addFilters(ui.getHandlers, conf).
|
Caution
|
FIXME Review JettyUtils.addFilters(ui.getHandlers, conf).
|
RegisterClusterManager Message
RegisterClusterManager(am: RpcEndpointRef)When RegisterClusterManager message arrives, the following INFO message is printed out to the logs:
INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as [am]The internal reference to the remote ApplicationMaster RPC Endpoint is set (to am).
If the internal shouldResetOnAmRegister flag is enabled, YarnSchedulerBackend is reset. It is disabled initially, so shouldResetOnAmRegister is enabled.
|
Note
|
shouldResetOnAmRegister controls whether to reset YarnSchedulerBackend when another RegisterClusterManager RPC message arrives that could be because the ApplicationManager failed and a new one was registered.
|
RetrieveLastAllocatedExecutorId
When RetrieveLastAllocatedExecutorId is received, YarnSchedulerEndpoint responds with the current value of currentExecutorIdCounter.
onDisconnected Callback
onDisconnected clears the internal reference to the remote ApplicationMaster RPC Endpoint (i.e. it sets it to None) if the remote address matches the reference’s.
|
Note
|
It is a callback method to be called when…FIXME |
You should see the following WARN message in the logs if that happens:
WARN ApplicationMaster has disassociated: [remoteAddress]onStop Callback
onStop shuts askAmThreadPool down immediately.
|
Note
|
onStop is a callback method to be called when…FIXME
|
Internal Reference to ApplicationMaster RPC Endpoint (amEndpoint variable)
amEndpoint is a reference to a remote ApplicationMaster RPC Endpoint.
It is set to the current ApplicationMaster RPC Endpoint when RegisterClusterManager arrives and cleared when the connection to the endpoint disconnects.