INFO YarnClusterScheduler: Created YarnClusterSchedulerYarnClusterScheduler - TaskScheduler for Cluster Deploy Mode
YarnClusterScheduler is the TaskScheduler for Spark on YARN in cluster deploy mode.
It is a custom YarnScheduler that makes sure that appropriate initialization of ApplicationMaster is performed, i.e. SparkContext is initialized and stopped.
While being created, you should see the following INFO message in the logs:
|
Tip
|
Enable Add the following line to Refer to Logging. |
postStartHook Callback
postStartHook calls ApplicationMaster.sparkContextInitialized before the parent’s postStartHook.
You should see the following INFO message in the logs:
INFO YarnClusterScheduler: YarnClusterScheduler.postStartHook doneStopping YarnClusterScheduler (stop method)
stop calls the parent’s stop followed by ApplicationMaster.sparkContextStopped.