Skip to content

Latest commit

 

History

History
30 lines (17 loc) · 1.37 KB

spark-LocalEndpoint.adoc

File metadata and controls

30 lines (17 loc) · 1.37 KB
Raw

LocalEndpoint

LocalEndpoint is the communication channel between Task Scheduler and LocalSchedulerBackend. It is a (thread-safe) RpcEndpoint that hosts an executor (with id driver and hostname localhost) for Spark local mode.

When a LocalEndpoint starts up (as part of Spark local’s initialization) it prints out the following INFO messages to the logs:

INFO Executor: Starting executor ID driver on host localhost
INFO Executor: Using REPL class URI: http://192.168.1.4:56131

reviveOffers Method

Caution
 FIXME

Creating LocalEndpoint Instance

Caution
 FIXME

RPC Messages

LocalEndpoint accepts the following RPC message types:

  • ReviveOffers (receive-only, non-blocking) - read Task Submission a.k.a. reviveOffers.

  • StatusUpdate (receive-only, non-blocking) that passes the message to TaskScheduler (using statusUpdate) and if the task’s status is finished, it revives offers (see ReviveOffers).

  • KillTask (receive-only, non-blocking) that kills the task that is currently running on the executor.

  • StopExecutor (receive-reply, blocking) that stops the executor.