StreamLines

An API for streaming files from remote locations one line at a time.

Background

Some applications run in production environments without a writable file system; usually this is a security measure. Futhermore, with the proliferation of container-based production environments, containers may not have access to tremendous amounts of memory. Thus, it can be impossible to read large files unless you read the file into memory in small doses. A common pattern is to use a line-delimited file like JSON Lines or a CSV and to read the file one line at a time in order to iterate over a dataset. This gem aims to provide an Enumerable interface for iterating over remote, line-delimited datasets.

Installation

Add this line to your application's Gemfile:

gem 'stream_lines'

And then execute:

$ bundle

Or install it yourself as:

$ gem install stream_lines

Usage

Reading

From a URL

url = 'https://my.remote.file/file.txt'
stream = StreamLines::Reading::Stream.new(url)

stream.each do |line|
  # Do something with the line of data (the line will be a String)
end

# A StreamLines::Reading::Stream object is Enumerable, so you can also use
# any Enumerable methods.

stream.each_slice(100) do |lines|
  # Do something with the 100 lines of data
end

Caveats

This library strives to provide streamed data via an Enumerable interface. In order to be memory-efficient, however, each time the stream is iterated over, a new GET request is made to fetch the data from its remote URL. For example,

url = 'https://my.remote.file/file.txt'
stream = StreamLines::Reading::Stream.new(url)
do_something_with_first_row(stream.first) # GET request made

stream.each do |line| # same GET request made
  # Do something with the line of data (the line will be a String)
end

makes two GET requests. The call to first makes a GET request to fetch the first row of data. The subsequent call to each makes the same GET request. To avoid unnecessary requests, I recommend a slightly different approach, which may not be intuitive but does make only one network request:

url = 'https://my.remote.file/file.txt'
stream = StreamLines::Reading::Stream.new(url)

stream.each_with_index do |line, i|
  do_something_with_first_row(line) if i.zero?
  # Do something with the line of data (the line will be a String)
end

CSVs

This gem provides first-class support for streaming CSVs from a remote URL.

url = 'https://my.remote.file/file.csv'
stream = StreamLines::Reading::CSV.new(url)

stream.each do |row|
  # each row will be an Array
end

# Supports most Ruby CSV options (see ignored options below)
stream = StreamLines::Reading::CSV.new(url, headers: true)

stream.each do |row|
  # each row will be a CSV::Row object that you can access like row['column_name']
end

Most options that you can pass to Ruby's CSV library are supported; however, the following options are explicitly ignored:

• return_headers
• header_converters
• skip_lines

I suspect that these options are not used terribly frequently, and each would require additional logic in the StreamLines::Reading::CSV#each method. Rather than attempting to implement sensible solutions for these options, I am choosing to explicitly ignore them until there is enough outcry to support them.

JSON Lines/Streaming JSON

This gem provides first-class support for streaming JSON lines from a remote URL.

url = 'https://my.remote.file/file.jsonl'
stream = StreamLines::Reading::JSONLines.new(url)

stream.each do |row|
  # each row will be an Hash
end

# Supports all Ruby JSON::parse options
stream = StreamLines::Reading::JSONLines.new(url, symbolize_names: true)

stream.each do |row|
  # each row will be a Hash
end

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install.

Releasing

After merging in the new functionality to the main branch:

git checkout main
git pull --prune
bundle exec rake version:bump:<major, minor, or patch>
bundle exec rubocop -a
git commit -a --amend
bundle exec rake release

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/jdlubrano/stream_lines. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the code of conduct.

License

The gem is available as open source under the terms of the MIT License.