Hadoop Fair Sojourn Protocol (HFSP)

The Hadoop Fair Sojourn Protocol Scheduler is a size-based scheduler for Hadoop. A presentation of the scheduler can be found here.

Compile HFSP

In order to compile HFSP you need Maven. From the top directory issue the following command:

$ mvn package -DskipTests

This will create two files in the directory /target:

• hfsp-scheduler-1.0.jar: a jar file containing the scheduler
• hfsp-scheduler.xml: a default configuration file

Use HFSP

Copy hfsp-scheduler-1.0.jar in your Hadoop directory. Optionally, add the configuration file for HFSP in the hadoop configuration directory.

Set HFSP as task scheduler in conf/mapred-site.xml:

<configuration>
	<property>
          <name>mapred.jobtracker.taskScheduler</name>        
          <value>org.apache.hadoop.mapred.HFSPScheduler</value>  
	</property>
</configuration>

Hadoop versions

HFSP has been developed for the current stable version of Hadoop 1.x, that is Hadoop 1.1.2.

Papers

• HFSP: Size-based scheduling for Hadoop
• Revisiting Size-Based Scheduling with Estimated Job Sizes

Contributors

• Mario Pastorelli ([email protected])
• Antonio Barbuzzi ([email protected])

Acknowledgements

The HFSP project is part of the BigFoot project