SnowFS - a fast, scalable version control file storage for graphic files

Overview

SnowFS is a lightweight multi-platform support library with a focus on binary file versioning. It is made for the graphics industry and was initially developed for Snowtrack.

Feature highlights

• Supports Branches

• Asynchronous File Hashing

• Project open to file-content awareness (e.g: *.psd, *.blend, *.c4d, ..)

• Super-fast-detection of modifications in large binaries

• Support for instant snapshots**

• Support for instant rollback**

• Support for files bigger >4TB

• Block-cloning and Copy-on-Write support for APFS and ReFS***

• Support for removing single versions and/or binaries

• Primarily I/O bound through libuv

• Feature XYZ made by you!

** If the underlying filesystem supports it (e.g. APFS, ReFS)

Why not Git/Git-LFS, libgit2, or SVN?

First and foremost, the implementations of Git - namely Git/Git-LFS and libgit2 are excellent implementations of version control systems. But due to their focus on the software development lifecycle they are not suitable to version binaries or graphic files. SnowFS addresses the technical challenges for graphic files by its core design.

Git/Git-LFS:

Advantages:

• Support on all major platforms
• Supported by hosting platforms like GitHub, GitLab, and BitBucket.
• Fast diff-operation for text-files

Disadvantages:

• (Without Git-LFS): Heavy cost with zipping, packing, and delta-compression for larger files
• If not properly tracked, binaries become accidentally part of "base" history
• Removing older commits is cumbersome due to Gits commit hashing integrity
• Complicated rewriting history procedure
• Issues with binaries >4GB on Windows as reported here, here, and here
• Slow in binary modification detection
• Git uses a restrictive license

libgit2

Advantages:

• Faster zipping, packing, and delta-compression than the reference implementation Git
• Supports custom backends

Disadvantages:

• No native support for Git-LFS without custom backends
• Custom backends break compatibility with Git

TypeScript / C++ backport

SnowFS is currently written in TypeScript. It is a great language to write powerful and performant I/O bound prototypes. There is a basic and experimental C/C++ backport, but we are looking for maintainers to get things finally rolling. If you have comments, ideas or recommendations, please let us know.

Examples

Code

You can find the best and up-to-date code examples in the test/ directory. Given below these are simply "Hello World!" examples to get you started.

import * as fse from "fs-extra";

import { join } from "path";
import { Index } from "./src";
import { Repository } from "./src/repository";

export async function main() {
  let repo: Repository;
  let index: Index;
  const repoPath = "/path/to/a/non/existing/directory";
  Repository.initExt(repoPath)
    .then((repoResult: Repository) => {
      return fse.copyFile("/path/to/texture.psd", join(repoPath, "texture.psd"));
    })
    .then(() => {
      index.addFiles(["texture.psd"]);
      return index.writeFiles();
    })
    .then(() => {
      return repo.createCommit(index, "This is my first commit");
    });
}

main();

Command line interface

The CLI of SnowFS offers some basic functionality and is subject to enhancements.

❗ Please note, due to ts-node the CLI has currently a very long warm-up phase. This will be fixed soon.

$ snow init foo
$ cp /path/to/texture.psd foo/texture.psd
$ cd foo
$ snow add .
$ snow commit -m "My first texture"
$ snow log
$ snow checkout -b MyNewBranch
$ snow log

Versioning

Starting with version 1.0.0 SnowFS follows the semantic versioning scheme. The API change and backward compatibility rules are those indicated by SemVer.

Licensing

SnowFS is licensed under the MIT license, please review the LICENSE file. Excluded from the license are images, artworks, and logos. Please file a request by mail, if you have any questions.

Community

• Discord
• Support
• Email

Other resources

The tests and benchmarks also serve as API specification and usage examples.

These resources are not handled by SnowFS maintainers and might be out of date. Please verify it before opening new issues.

Build Instructions

To build SnowFS you need a current version of node.js, To build with node:

$ git clone https://github.com/Snowtrack/snowfs.git
$ cd snowfs.git
$ npm install
$ npm run ava

Running benchmarks

We have also implemented a comparison benchmark between SnowFS vs. git. The benchmarks can be executed (after building) with the following command:

$ npm run benchmarks

Supported Platforms

Currently, Windows, macOS, and Linux are supported. SnowFS works on plain filesystems like FAT, NTFS, HFS+ and has extended support for APFS and ReFS*.

How can I contribute?

See the guidelines for contributing.