Spark Basics - Submitting Spark Jobs

Spark Basics - Spark Job Submission#

Content organized from:

Spark Architecture and Job Execution Process

Application: A Spark application written by the user, which includes code for a Driver function and Executor code that runs on multiple nodes distributed across the cluster.
Driver: The Driver runs the main() function of the above Application and creates the SparkContext to prepare the runtime environment for the Spark application. In Spark, the SparkContext is responsible for communicating with the ClusterManager, applying for resources, task allocation, monitoring, etc.; after the Executor part has finished running, the Driver is responsible for shutting down the SparkContext. Typically, SparkContext represents the Driver.
Worker: Any node in the cluster that can run Application code, similar to the NodeManager node in YARN. In Standalone mode, it refers to Worker nodes configured through the Slave file, while in Spark on Yarn mode, it refers to NodeManager nodes.
Executor: A process that runs on Worker nodes for the Application, responsible for executing Tasks and storing data in memory or on disk. Each Application has its own independent set of Executors.
Cluster Manager: Refers to the external service that acquires resources in the cluster, currently including:
1. Standalone: Native resource management of Spark, with resource allocation managed by the Master.
2. Hadoop Yarn: Resource allocation managed by the ResourceManager in YARN.

Two core components: DAGScheduler and TaskScheduler

For details, see: Spark Advanced - Spark Scheduling System

In local mode, there is no concept of master+worker.
In local mode, it is equivalent to starting a local process, simulating the execution of a Spark job in a Spark cluster, where one or more executor threads correspond to the process and begin execution, including job scheduling and task allocation.

In standalone mode submission, the master needs to be set to spark://master_ip, for example, spark://192.168.75.101:7077.

Process Description

After the client starts, it directly runs the user program, initiating Driver-related tasks: DAGScheduler and BlockManagerMaster, etc.
The Driver of the client registers with the Master.
The Master will also instruct the Worker to start the Executor. The Worker creates an ExecutorRunner thread, which will start the ExecutorBackend process.
After the ExecutorBackend starts, it registers with the Driver's SchedulerBackend. The Driver's DAGScheduler parses the job and generates the corresponding stages, with tasks in each stage allocated to Executors for execution by the TaskScheduler.
The job ends after all stages are completed.

The client submits the job to the master.
The Master selects a Worker node to start the Driver, i.e., SchedulerBackend. The Worker creates a DriverRunner thread, and the DriverRunner starts the SchedulerBackend process.
The Master instructs the remaining Workers to start Executors, i.e., ExecutorBackend. The Worker creates an ExecutorRunner thread, which will start the ExecutorBackend process.
After the ExecutorBackend starts, it registers with the Driver's SchedulerBackend. The SchedulerBackend process includes the DAGScheduler, which generates an execution plan based on the user program and schedules execution. For each stage's task, it will be stored in the TaskScheduler, and when the ExecutorBackend reports to the SchedulerBackend, it schedules the tasks in the TaskScheduler for execution by the ExecutorBackend.

In client mode, the Driver starts on the machine where the spark-submit script is run.
In cluster mode, the Driver is randomly assigned to start on a Worker process through the master process.

The Spark Yarn Client requests to start the Application Master from YARN's ResourceManager.
After receiving the request, the ResourceManager selects a NodeManager in the cluster to allocate the first Container for the application, requiring it to start the ApplicationMaster in this Container. The difference from YARN-Cluster is that this ApplicationMaster does not run SparkContext, but only communicates with SparkContext for resource allocation.
After the SparkContext in the Client is initialized, it establishes communication with the ApplicationMaster, registers with the ResourceManager, and requests resources (Containers) based on task information.
After the ApplicationMaster obtains resources (i.e., Containers), it communicates with the corresponding NodeManager, requesting it to start the CoarseGrainedExecutorBackend in the obtained Container. After the CoarseGrainedExecutorBackend starts, it registers with the SparkContext in the Client and requests Tasks.
The SparkContext in the Client allocates Tasks to the CoarseGrainedExecutorBackend for execution, and the CoarseGrainedExecutorBackend runs the Tasks and reports the execution status and progress to the Driver.
After the application completes, the SparkContext in the Client requests deregistration from the ResourceManager and shuts itself down.

In YARN-Cluster mode, when a user submits an application to YARN, YARN runs the application in two stages:

The first stage is to start the Spark Driver as an ApplicationMaster in the YARN cluster.
The second stage is for the ApplicationMaster to create the application, request resources from the ResourceManager, and start Executors to run Tasks while monitoring the entire execution process until completion.

In YARN-Cluster mode, the Driver runs in the AM (Application Master), which is responsible for requesting resources from YARN and supervising the job's execution status. Once the user submits the job, the Client can be closed, and the job will continue running on YARN, making YARN-Cluster mode unsuitable for interactive jobs.
In YARN-Client mode, the Application Master only requests Executors from YARN, and the Client communicates with the requested Containers to schedule their work, meaning the Client cannot leave. The Driver runs on the client machine that submitted the Spark job, allowing real-time access to detailed log information, facilitating error tracking and troubleshooting, and is used for testing.