README

NAME
	br - bashreduce, map/reduce in bash

SYNOPSIS
	br [-h '<host>[ <host>][...]'] [-b] [-f]
		[-m <map>] [-r <reduce>] [-M <merge>]
		[-i <input>] [-o <output>] [-c <column>] [-S <sort-mem-MB>]
		[-t <tmp>] [-?]

DESCRIPTION
	br implements a map/reduce framework that allows the map and reduce
	jobs to be constructed from standard Unix tools.  It can operate on
	data piped into it or on files known to exist on the worker nodes.

INSTALLATION
	br can run out-of-the box but for convenience, put it somewhere on
	your PATH.  The brutils come with a Makefile that will install
	these tools, probably to /usr/local/bin.  If you leave these tools
	in place relative to br, they'll be picked up and used from there.

	You'll need passwordless ssh access to each machine you wish to
	use as a worker (even localhost, for now).

OPTIONS
	-h	The list of hosts to be used, specified as a space-
		delimited list.  Specifying the same host multiple
		times is allowed and is useful for taking explicit
		advantage of multiple cores.

	-b	The list of hosts to be used is obtained from the
		input file's extended attributes, if the input file
		is stored on BeeFS [2].  This option automatically
    enables the -f option. 

	-f	Only pass filenames from the master to the workers,
		under the assumption that they all have a mirror of
		the data set.  This is equivalent to prefixing your
		map program with "xargs cat |".

	-m	The map program, which should expect initial data on
		stdin and produce the intermediate data on stdout.
		The stderr from this program will be logged [1].

	-r	The reduce program, which should expect intermediate
		data on stdin and produce the final data on stdout.
		The stderr from this program will be logged [1].

	-M	The merge program, which should expect each node's
		final data on stdin and produce the unified final
		data on stdout.  The stderr from this program will
		be logged [1].

	-i	A file or directory to serve as input.  If it is a
		file, this is equivalent to piping that file into
		br.  If it is a directory, it is equivalent to
		piping the names of every file in that directory
		into br.

	-o	The local file where output will be stored.
		Defaults to stdout.

	-c	The column used by sort to produce the final output.
		Defaults to 1.

	-S	This is directly the -S option from sort(1).  It defaults
		to 256M.

	-t	The temporary directory for storing br files.  All of
		these files will be prefixed with "br_" and, with the
		exception of br_stderr, will be cleaned up upon
		successful exit.  Defaults to /tmp.

	-?	Shows a help message and details about the arguments.

FILES
	All of these files are in the tmp directory specified by the user,
	which defaults to /tmp.

	br_job_*
		The master uses named pipes to move data (or filenames)
		to the worker(s).  These pipes are stored here.  This
		directory is removed at the end of the job.

	br_node_*
		Each node uses named pipes to buffer data.  These
		are stored here and do not overlap, even if a
		node is specified twice in the host list.  These
		directories are removed at the end of the job.

	br_stderr
		This file is cleared at the beginning of each
		job.  It records stderr from most commands run
		by br.  This is currently a point of contention
		between jobs.  This file is NOT removed at the
		end of the job.

NOTES
	[1]	Because you can specify a pipeline for any or all of the map,
		reduce and merge steps, br can only reliably log stderr from
		the last step in those pipelines.

	[2]	The extended attributes are obtained via getfattr tool.
		If you don't have getfattr installed yet, do:
		$> sudo apt-get install attr

AUTHORS
	Erik Frey <erik@fawx.com>
	Richard Crowley <r@rcrowley.org>
	Jonhnny Weslley <jw@jonhnnyweslley.net>

SEE ALSO
	<http://github.com/erikfrey/bashreduce>
	<http://github.com/rcrowley/bashreduce>
	<http://github.com/jweslley/bashreduce>

LICENSE
	The goodwill of Erik Frey.