spark-blocktransferservice.adoc

BlockTransferService — Pluggable Block Transfers

BlockTransferService is a contract for ShuffleClients that can fetch and upload blocks synchronously or asynchronously.

Note	BlockTransferService is a private[spark] abstract class.

Note	NettyBlockTransferService is the only available implementation of BlockTransferService Contract.

Note	BlockTransferService was introduced in SPARK-3019 Pluggable block transfer interface (BlockTransferService) and is available since Spark 1.2.0.

BlockTransferService Contract

Every BlockTransferService offers the following:

• init that accepts BlockDataManager for storing or fetching blocks. It is assumed that the method is called before a BlockTransferService service is considered fully operational.

  init(blockDataManager: BlockDataManager): Unit

• port the service listens to.

  port: Int

• hostName the service listens to.

  hostName: String

• uploadBlock to upload a block (of ManagedBuffer identified by BlockId) to a remote hostname and port.

  uploadBlock(
    hostname: String,
    port: Int,
    execId: String,
    blockId: BlockId,
    blockData: ManagedBuffer,
    level: StorageLevel,
    classTag: ClassTag[_]): Future[Unit]

• Synchronous (and hence blocking) fetchBlockSync to fetch one block blockId (that corresponds to the ShuffleClient parent’s asynchronous fetchBlocks).

  fetchBlockSync(
    host: String,
    port: Int,
    execId: String,
    blockId: String): ManagedBuffer

  fetchBlockSync is a mere wrapper around fetchBlocks to fetch one blockId block that waits until the fetch finishes.

uploadBlockSync Method

uploadBlockSync(
  hostname: String,
  port: Int,
  execId: String,
  blockId: BlockId,
  blockData: ManagedBuffer,
  level: StorageLevel,
  classTag: ClassTag[_]): Unit

uploadBlockSync is a mere blocking wrapper around uploadBlock that waits until the upload finishes.

Note	uploadBlockSync is only executed when BlockManager replicates a block to another node(s) (i.e. when a replication level is greater than 1).