Skip to content

Latest commit

 

History

History
63 lines (50 loc) · 1.5 KB

README.md

File metadata and controls

63 lines (50 loc) · 1.5 KB
Raw

python gridfs fuse

A FUSE wrapper around MongoDB gridfs using python and llfuse.

Usage

gridfs_fuse --mongodb-uri="mongodb://127.0.0.1:27017" --database="gridfs_fuse" --mount-point="/mnt/gridfs_fuse"

Requirements

  • pymongo
  • llfuse

Operations supported

  • create/list/delete directories => folder support.
  • read files.
  • delete files.
  • open and write once (like HDFS).
  • rename

Operations not supported

  • modify an existing file.
  • resize an existing file.
  • hardlink
  • symlink
  • statfs

Performance

Setup

  • AWS d2.xlarge machine.
    • 4 @ 2.40Ghz (E5-2676)
    • 30 gigabyte RAM
  • filesystem: ext4
  • block device: three instance storage disks combined with lvm.
lvcreate -L 3T -n mongo -i 3 -I 4096 ax /dev/xvdb /dev/xvdc /dev/xvdd
  • mongodb 3.0.1
  • mongodb storage engine WiredTiger
  • mongodb compression: snappy
  • mongodb cache size: 10 gigabyte

Results

  • sequential write performance: ~46 MB/s
  • sequential read performance: ~90 MB/s

Write performance was tested by copying 124 files, each having a size of 9 gigabytes and different content. Compression factor was about factor three. Files were copied one by one => no parallel execution.

Read performance was tested by randomly picking 10 files out of the 124. Files were read one by one => no parallel execution.

# Simple illustration of the commands used (not the full script).

# Write
pv -pr /tmp/big_file${file_number} /mnt/gridfs_fuse/

# Read
pv -pr /mnt/gridfs_fuse${file_number} > /dev/null