STEC EnhanceIO SSD Caching Software
				25th December, 2012


1. WHAT IS ENHANCEIO?

	EnhanceIO driver is based on EnhanceIO SSD caching software product 
	developed by STEC Inc. EnhanceIO was derived from Facebook's open source
	Flashcache project. EnhanceIO uses SSDs as cache devices for
	traditional rotating hard disk drives (referred to as source volumes
	throughout this document).

	EnhanceIO can work with any block device, be it an entire physical
	disk, an individual disk partition,  a RAIDed DAS device, a SAN volume,
	a device mapper volume or a software RAID (md) device.

	The source volume to SSD mapping is a set-associative mapping based on
	the source volume sector number with a default set size
	(aka associativity) of 512 blocks and a default block size of 4 KB.
	Partial cache blocks are not used. The default value of 4 KB is chosen
	because it is the common I/O block size of most storage systems.  With
	these default values, each cache set is 2 MB (512 * 4 KB).  Therefore,
	a 400 GB SSD will have a little less than 200,000 cache sets because a
	little space is used for storing the meta data on the SSD.

	EnhanceIO supports three caching modes: read-only, write-through, and
	write-back and three cache replacement policies: random, FIFO, and LRU.

	Read-only caching mode causes EnhanceIO to direct write IO requests only
	to HDD. Read IO requests are issued to HDD and the data read from HDD is
	stored on SSD. Subsequent Read requests for the same blocks are carried
	out from SSD, thus reducing their latency by a substantial amount. 

	In Write-through mode - reads are handled similar to Read-only mode.
	Write-through mode causes EnhanceIO to write application data to both
	HDD and SSD. Subsequent reads of the same data benefit because they can
	be served from SSD.

	Write-back improves write latency by writing application requested data
	only to SSD. This data, referred to as dirty data, is copied later to
	HDD asynchronously. Reads are handled similar to Read-only and
	Write-through modes.

2. WHAT HAS ENHANCEIO CHANGED TO FLASHCACHE?

2.1. A new write-back engine

	The write-back engine in EnhanceiO has been designed from scratch.
	Several optimizations have been done. IO completion guarantees have
	been improved. We have defined limits to let a user control the amount
	of dirty data in a cache. Clean-up of dirty data is stopped by default
	under a high load; this can be overridden if required. A user can
	control the extent to which a single cache set can be filled with dirty
	data. A background thread cleans-up dirty data at regular intervals.
	Clean-up is also done at regular intevals by identifying cache sets
	which have been written least recently.

2.2. Transparent cache

	EnhanceIO does not use device mapper. This enables creation and
	deletion of caches while a source volume is being used. It's possible
	to either create or delete cache while a partition is mounted.

	EnhanceIO also supports creation of a cache for a device which contains
	partitions. With this feature it's possible to create a cache without
	worrying about having to create several SSD partitions and many
	separate caches.


2.3. Large I/O Support

	Unlike Flashcache, EnhanceIO does not cause source volume I/O requests
	to be split into cache block size pieces. For the typical SSD cache
	block size of 4 KB, this means that a write I/O request size of, say,
	64 KB to the source volume is not split into 16 individual requests of
	4 KB each. This is a performance improvement over Flashcache. IO
	codepaths have been substantially modified for this improvement.

2.4. Small Memory Footprint

	Through a special compression algorithm, the meta data RAM usage has
	been reduced to only 4 bytes for each SSD cache block (versus 16 bytes
	in Flashcache).  Since the most typical SSD cache block size is 4 KB,
	this means that RAM usage is 0.1% (1/1000) of SSD capacity.
	For example, for a 400 GB SSD, EnhanceIO will need only 400 MB to keep
	all meta data in RAM.

	For an SSD cache block size of 8 KB, RAM usage is 0.05% (1/2000) of SSD
	capacity.

	The compression algorithm needs at least 32,768 cache sets
	(i.e., 16 bits to encode the set number). If the SSD capacity is small
	and there are not at least 32,768 cache sets, EnhanceIO uses 8 bytes of
	RAM for each SSD cache block. In this case, RAM usage is 0.2% (2/1000)
	of SSD capacity for a cache block size of 4K.

2.5. Loadable Replacement Policies

	Since the SSD cache size is typically 10%-20% of the source volume
	size, the set-associative nature of EnhanceIO necessitates cache
	block replacement.

	The main EnhanceIO kernel module that implements the caching engine
	uses a random (actually, almost like round-robin) replacement policy
	that does not require any additional RAM and has the least CPU
	overhead.  However, there are two additional kernel modules that
	implement FIFO and LRU replacement policies.  FIFO is the default cache
	replacement policy because it uses less RAM than LRU.  The FIFO and LRU
	kernel modules are independent of each other and do not have to be
	loaded if they are not needed.

	Since the replacement policy modules do not consume much RAM when not
	used, both modules are typically loaded after the main caching engine
	is loaded. RAM is used only after a cache has been instantiated to use
	either the FIFO or the LRU replacement policy.

	Please note that the RAM used for replacement policies is in addition
	to the RAM used for meta data (mentioned in Section 2.1).  The table
	below shows how	much RAM each cache replacement policy uses:

		POLICY	RAM USAGE
		------	---------
		Random	0
		FIFO	4 bytes per cache set
		LRU	4 bytes per cache set + 4 bytes per cache block

2.6. Optimal Alignment of Data Blocks on SSD

	EnhanceIO writes all meta data and data blocks on 4K-aligned blocks
	on the SSD. This minimizes write amplification and flash wear.
	It also improves performance.

2.7. Improved device failure handling

	Failure of an SSD device in read-only and write-through modes is
	handled gracefully by allowing I/O to continue to/from the
	source volume. An application may notice a drop in performance but it
	will not receive any I/O errors.

	Failure of an SSD device in write-back mode obviously results in the
	loss of dirty blocks in the cache. To guard against this data loss, two
	SSD devices can be mirrored via RAID 1.

	EnhanceIO identifies device failures based on error codes. Depending on
	whether the failure is likely to be intermittent or permanent, it takes
	the best suited action.

2.8. Coding optimizations

	Several coding optizations have been done to reduce CPU usage.  These
	include removing queues which are not required for write-through and
	read-only cache modes, splitting of a single large spinlock, and more.
	Most of the code paths in flashcache have been substantially
	restructured.

2.9 Sequential I/O bypass

	EnhanceIO has removed the bypass of sequential IO available in flashcache.
	The sequential detection logic has a limited use case, espescially in a
	reasonably multithreaded scenario.


3. EnhanceIO usage

3.1. Cache creation, deletion and editing properties

	eio_cli utility is used for creating and deleting caches and editing
	their properties. Manpage for this utility eio_cli(8) provides more
	information.

3.2. Making a cache configuration persistent
	It's essential that a cache be resumed before any applications or a
	filesystem use the source volume during a bootup. If a cache is enabled
	after a source volume is written to, stale data may be present in the
	cache. It may cause data corruption. The document Persistent.txt
	describes how to enable a cache during bootup using udev scripts.

	In case an SSD does not come up during a bootup, it's ok to allow read
	and write access to HDD only in the case of a Write-through or a
	read-only cache. A cache should be created again when SSD becomes
	available. If a previous cache configuration is resumed, it may cause
	stale data to be read.

3.3. Using a Write-back cache
	It's absolutely necessary to make a Write-back cache configuration
	persistent. This is required particularly in the case of an OS crash or
	a power failure.  A Write-back cache may contain dirty blocks which
	haven't been written to HDD yet. Reading the source volume without
	enabling the cache will cause incorrect data to be read.

	In case an SSD does not come up during a bootup, access to HDD should
	stopped. It should be enabled only after SSD comes-up and a cache is
	enabled.
	
	Write-back cache needs to perform clean operation in order to flush the
	dirty data to the source device(HDD). The clean can be either trigerred
	by the user or automatically initiated, based on preconfigured
	thresholds. These thresholds are described below. They can be set using 
	sysctl calls.

	a) Dirty high threshold (%) : The upper limit on percentage of dirty
	   blocks in the entire cache.
	b) Dirty low threshold (%) : The lower limit on percentage of dirty
	   blocks in the entire cache.
	c) Dirty set high threshold (%) : The upper limit on percentage of dirty
	   blocks in a set.
	d) Dirty set low threshold (%) : The lower limit on percentage of dirty
	   blocks in a set.	
	e) Automatic clean-up threshold : An automatic clean-up of the cache
	   will occur only if the number of outstanding I/O requests from the
	   HDD is below the threshold.
	f) Time based clean-up interval (minutes) : This option allows you to
	   specify an interval between each clean-up process.

	Clean is trigerred when one of the upper thresholds or time based clean 
	threshold is met and stops when all the lower thresholds are met.  


4. ACKNOWLEDGEMENTS

	STEC acknowledges Facebook and in particular Mohan Srinivasan
	for the design, development, and release of Flashcache as an
	open source project.

	Flashcache, in turn, is based on DM-Cache by Ming Zhao.