V1.6.0 adam fix v2

CHANGELOG.md

@@ -2,10 +2,98 @@
 All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
-and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
+and this project adheres to [Serialization Versioning](VERSIONING.md).
 
 ## [Unreleased]
 
+## [1.6.0] - 2019-07-01
+
+This is a release including almost 200 commits, so expect many changes - some of them
+not backward compatible.
+
+Regarding backward compatibility in Badger versions, you might be interested on reading
+[VERSIONING.md](VERSIONING.md).
+
+_Note_: The hashes in parentheses correspond to the commits that impacted the given feature.
+
+### New APIs
+
+- badger.DB
+  - DropPrefix (291295e)
+  - Flatten (7e41bba)
+  - KeySplits (4751ef1)
+  - MaxBatchCount (b65e2a3)
+  - MaxBatchSize (b65e2a3)
+  - PrintKeyValueHistogram (fd59907)
+  - Subscribe (26128a7)
+  - Sync (851e462)
+
+- badger.DefaultOptions() and badger.LSMOnlyOptions() (91ce687)
+  - badger.Options.WithX methods
+
+- badger.Entry (e9447c9)
+  - NewEntry
+  - WithMeta
+  - WithDiscard
+  - WithTTL 
+
+- badger.Item
+  - KeySize (fd59907)
+  - ValueSize (5242a99)
+
+- badger.IteratorOptions
+  - PickTable (7d46029, 49a49e3)
+  - Prefix (7d46029)
+
+- badger.Logger (fbb2778)
+
+- badger.Options
+  - CompactL0OnClose (7e41bba)
+  - Logger (3f66663)
+  - LogRotatesToFlush (2237832)
+
+- badger.Stream (14cbd89, 3258067)
+- badger.StreamWriter (7116e16)
+- badger.TableInfo.KeyCount (fd59907)
+- badger.TableManifest (2017987)
+- badger.Tx.NewKeyIterator (49a49e3)
+- badger.WriteBatch (6daccf9, 7e78e80)
+
+### Modified APIs
+
+#### Breaking changes:
+
+- badger.DefaultOptions and badger.LSMOnlyOptions are now functions rather than variables (91ce687)
+- badger.Item.Value now receives a function that returns an error (439fd46)
+- badger.Txn.Commit doesn't receive any params now (6daccf9)
+- badger.DB.Tables now receives a boolean (76b5341)
+
+#### Not breaking changes:
+
+- badger.LSMOptions changed values (799c33f)
+- badger.DB.NewIterator now allows multiple iterators per RO txn (41d9656)
+- badger.Options.TableLoadingMode's new default is options.MemoryMap (6b97bac)
+
+### Removed APIs
+
+- badger.ManagedDB (d22c0e8)
+- badger.Options.DoNotCompact (7e41bba)
+- badger.Txn.SetWithX (e9447c9)
+
+### Tools:
+
+- badger bank disect (13db058)
+- badger bank test (13db058) --mmap (03870e3)
+- badger fill (7e41bba)
+- badger flatten (7e41bba)
+- badger info --histogram (fd59907) --history --lookup --show-keys --show-meta --with-prefix (09e9b63) --show-internal (fb2eed9)
+- badger benchmark read (239041e)
+- badger benchmark write (6d3b67d)
+
+## [1.5.5] - 2019-06-20
+
+* Introduce support for Go Modules
+
 ## [1.5.3] - 2018-07-11
 Bug Fixes:
 * Fix a panic caused due to item.vptr not copying over vs.Value, when looking
@@ -87,7 +175,9 @@ Bug fix:
 ## [1.0.1] - 2017-11-06
 * Fix an uint16 overflow when resizing key slice
 
-[Unreleased]: https://github.com/dgraph-io/badger/compare/v1.5.3...HEAD
+[Unreleased]: https://github.com/dgraph-io/badger/compare/v1.6.0...HEAD
+[1.6.0]: https://github.com/dgraph-io/badger/compare/v1.5.5...v1.6.0
+[1.5.5]: https://github.com/dgraph-io/badger/compare/v1.5.3...v1.5.5
 [1.5.3]: https://github.com/dgraph-io/badger/compare/v1.5.2...v1.5.3
 [1.5.2]: https://github.com/dgraph-io/badger/compare/v1.5.1...v1.5.2
 [1.5.1]: https://github.com/dgraph-io/badger/compare/v1.5.0...v1.5.1

README.md

@@ -6,7 +6,7 @@ BadgerDB is an embeddable, persistent and fast key-value (KV) database
 written in pure Go. It's meant to be a performant alternative to non-Go-based
 key-value stores like [RocksDB](https://github.com/facebook/rocksdb).
 
-## Project Status [Oct 27, 2018]
+## Project Status [Jun 26, 2019]
 
 Badger is stable and is being used to serve data sets worth hundreds of
 terabytes. Badger supports concurrent ACID transactions with serializable
@@ -15,14 +15,20 @@ snapshot isolation (SSI) guarantees. A Jepsen-style bank test runs nightly for
 Badger has also been tested to work with filesystem level anomalies, to ensure
 persistence and consistency.
 
-Badger v1.0 was released in Nov 2017, with a Badger v2.0 release coming up in a
-few months. The [Changelog] is kept fairly up-to-date.
+Badger v1.0 was released in Nov 2017, and the latest version that is data-compatible
+with v1.0 is v1.6.0.
+
+Badger v2.0, a new release coming up very soon will use a new storage format which won't
+be compatible with all of the v1.x. The [Changelog] is kept fairly up-to-date.
+
+For more details on our version naming schema please read [Choosing a version](#choosing-a-version).
 
 [Changelog]:https://github.com/dgraph-io/badger/blob/master/CHANGELOG.md
 
 ## Table of Contents
  * [Getting Started](#getting-started)
     + [Installing](#installing)
+      - [Choosing a version](#choosing-a-version)
     + [Opening a database](#opening-a-database)
     + [Transactions](#transactions)
       - [Read-only transactions](#read-only-transactions)
@@ -61,6 +67,27 @@ $ go get github.com/dgraph-io/badger/...
 This will retrieve the library and install the `badger` command line
 utility into your `$GOBIN` path.
 
+#### Choosing a version
+
+BadgerDB is a pretty special package from the point of view that the most important change we can
+make to it is not on its API but rather on how data is stored on disk.
+
+This is why we follow a version naming schema that differs from Semantic Versioning.
+
+- New major versions are released when the data format on disk changes in an incompatible way.
+- New minor versions are released whenever the API changes but data compatibility is maintained.
+ Note that the changes on the API could be backward-incompatible - unlike Semantic Versioning.
+- New patch versions are released when there's no changes to the data format nor the API.
+
+Following these rules:
+
+- v1.5.0 and v1.6.0 can be used on top of the same files without any concerns, as their major
+ version is the same, therefore the data format on disk is compatible.
+- v1.6.0 and v2.0.0 are data incompatible as their major version implies, so files created with
+ v1.6.0 will need to be converted into the new format before they can be used by v2.0.0.
+
+For a longer explanation on the reasons behind using a new versioning naming schema, you can read
+[VERSIONING.md](VERSIONING.md).
 
 ### Opening a database
 The top-level object in Badger is a `DB`. It represents multiple files on disk
@@ -82,10 +109,7 @@ import (
 func main() {
   // Open the Badger database located in the /tmp/badger directory.
   // It will be created if it doesn't exist.
-  opts := badger.DefaultOptions
-  opts.Dir = "/tmp/badger"
-  opts.ValueDir = "/tmp/badger"
-  db, err := badger.Open(opts)
+  db, err := badger.Open(badger.DefaultOptions("tmp/badger"))
   if err != nil {
 	  log.Fatal(err)
   }

VERSIONING.md

@@ -0,0 +1,47 @@
+# Serialization Versioning: Semantic Versioning for databases
+
+Semantic Versioning, commonly known as SemVer, is a great idea that has been very widely adopted as
+a way to decide how to name software versions. The whole concept is very well summarized on
+semver.org with the following lines:
+
+> Given a version number MAJOR.MINOR.PATCH, increment the:
+> 
+> 1. MAJOR version when you make incompatible API changes,
+> 2. MINOR version when you add functionality in a backwards-compatible manner, and
+> 3. PATCH version when you make backwards-compatible bug fixes.
+> 
+> Additional labels for pre-release and build metadata are available as extensions to the
+> MAJOR.MINOR.PATCH format.
+
+Unfortunately, API changes are not the most important changes for libraries that serialize data for
+later consumption. For these libraries, such as BadgerDB, changes to the API are much easier to
+handle than change to the data format used to store data on disk.
+
+## Serialization Version specification
+
+Serialization Versioning, like Semantic Versioning, uses 3 numbers and also calls them
+MAJOR.MINOR.PATCH, but the semantics of the numbers are slightly modified:
+
+Given a version number MAJOR.MINOR.PATCH, increment the:
+
+- MAJOR version when you make changes that require a transformation of the dataset before it can be
+used again.
+- MINOR version when old datasets are still readable but the API might have changed in
+backwards-compatible or incompatible ways.
+- PATCH version when you make backwards-compatible bug fixes.
+
+Additional labels for pre-release and build metadata are available as extensions to the
+MAJOR.MINOR.PATCH format.
+
+Following this naming strategy, migration from v1.x to v2.x requires a migration strategy for your
+existing dataset, and as such has to be carefully planned. Migrations in between different minor
+versions (e.g. v1.5.x and v1.6.x) might break your build, as the API *might* have changed, but once
+your code compiles there's no need for any data migration. Lastly, changes in between two different
+patch versions should never break your build or dataset.
+
+For more background on our decision to adopt Serialization Versioning, read the blog post
+[Semantic Versioning, Go Modules, and Databases][blog] and the original proposal on
+[this comment on Dgraph's Discuss forum][discuss].
+
+[blog]: https://blog.dgraph.io/post/serialization-versioning/
+[discuss]: https://discuss.dgraph.io/t/go-modules-on-badger-and-dgraph/4662/7

backup.go

@@ -69,28 +69,10 @@ func (stream *Stream) Backup(w io.Writer, since uint64) (uint64, error) {
 				}
 			}
 
-			// Convert key and/or values to a different format if requested before writing
-			// the key-value pair.
-			keyCopy := item.KeyCopy(nil)
-			if stream.db.opt.BackupKeyFn != nil {
-				var err error
-				keyCopy, err = stream.db.opt.BackupKeyFn(keyCopy)
-				if err != nil {
-					return nil, err
-				}
-			}
-			if stream.db.opt.BackupValueFn != nil {
-				var err error
-				valCopy, err = stream.db.opt.BackupValueFn(valCopy)
-				if err != nil {
-					return nil, err
-				}
-			}
-
 			// clear txn bits
 			meta := item.meta &^ (bitTxn | bitFinTxn)
 			kv := &pb.KV{
-				Key:       keyCopy,
+				Key:       item.KeyCopy(nil),
 				Value:     valCopy,
 				UserMeta:  []byte{item.UserMeta()},
 				Version:   item.Version(),
@@ -104,7 +86,7 @@ func (stream *Stream) Backup(w io.Writer, since uint64) (uint64, error) {
 				// If we need to discard earlier versions of this item, add a delete
 				// marker just below the current version.
 				list.Kv = append(list.Kv, &pb.KV{
-					Key:     keyCopy,
+					Key:     item.KeyCopy(nil),
 					Version: item.Version() - 1,
 					Meta:    []byte{bitDelete},
 				})
@@ -145,20 +127,23 @@ func writeTo(list *pb.KVList, w io.Writer) error {
 	return err
 }
 
-type loader struct {
+// KVLoader is used to write KVList objects in to badger. It can be used to restore a backup.
+type KVLoader struct {
 	db       *DB
 	throttle *y.Throttle
 	entries  []*Entry
 }
 
-func (db *DB) newLoader(maxPendingWrites int) *loader {
-	return &loader{
+// NewKVLoader returns a new instance of KVLoader.
+func (db *DB) NewKVLoader(maxPendingWrites int) *KVLoader {
+	return &KVLoader{
 		db:       db,
 		throttle: y.NewThrottle(maxPendingWrites),
 	}
 }
 
-func (l *loader) set(kv *pb.KV) error {
+// Set writes the key-value pair to the database.
+func (l *KVLoader) Set(kv *pb.KV) error {
 	var userMeta, meta byte
 	if len(kv.UserMeta) > 0 {
 		userMeta = kv.UserMeta[0]
@@ -180,7 +165,7 @@ func (l *loader) set(kv *pb.KV) error {
 	return nil
 }
 
-func (l *loader) send() error {
+func (l *KVLoader) send() error {
 	if err := l.throttle.Do(); err != nil {
 		return err
 	}
@@ -194,7 +179,8 @@ func (l *loader) send() error {
 	return nil
 }
 
-func (l *loader) finish() error {
+// Finish is meant to be called after all the key-value pairs have been loaded.
+func (l *KVLoader) Finish() error {
 	if len(l.entries) > 0 {
 		if err := l.send(); err != nil {
 			return err
@@ -205,15 +191,16 @@ func (l *loader) finish() error {
 
 // Load reads a protobuf-encoded list of all entries from a reader and writes
 // them to the database. This can be used to restore the database from a backup
-// made by calling DB.Backup().
+// made by calling DB.Backup(). If more complex logic is needed to restore a badger
+// backup, the KVLoader interface should be used instead.
 //
 // DB.Load() should be called on a database that is not running any other
 // concurrent transactions while it is running.
 func (db *DB) Load(r io.Reader, maxPendingWrites int) error {
 	br := bufio.NewReaderSize(r, 16<<10)
 	unmarshalBuf := make([]byte, 1<<10)
 
-	ldr := db.newLoader(maxPendingWrites)
+	ldr := db.NewKVLoader(maxPendingWrites)
 	for {
 		var sz uint64
 		err := binary.Read(br, binary.LittleEndian, &sz)
@@ -237,24 +224,7 @@ func (db *DB) Load(r io.Reader, maxPendingWrites int) error {
 		}
 
 		for _, kv := range list.Kv {
-			// If the keys or values were transformed before backup, reverse those
-			// changes before restoring the key-value pair.
-			if db.opt.RestoreKeyFn != nil {
-				var err error
-				kv.Key, err = db.opt.RestoreKeyFn(kv.Key)
-				if err != nil {
-					return err
-				}
-			}
-			if db.opt.RestoreValueFn != nil {
-				var err error
-				kv.Value, err = db.opt.RestoreValueFn(kv.Value)
-				if err != nil {
-					return err
-				}
-			}
-
-			if err := ldr.set(kv); err != nil {
+			if err := ldr.Set(kv); err != nil {
 				return err
 			}
 
@@ -266,7 +236,7 @@ func (db *DB) Load(r io.Reader, maxPendingWrites int) error {
 		}
 	}
 
-	if err := ldr.finish(); err != nil {
+	if err := ldr.Finish(); err != nil {
 		return err
 	}
 	db.orc.txnMark.Done(db.orc.nextTxnTs - 1)

badger/cmd/bank.go

@@ -181,7 +181,7 @@ func diff(a, b []account) string {
 	return buf.String()
 }
 
-var errFailure = errors.New("Found an balance mismatch. Test failed.")
+var errFailure = errors.New("test failed due to balance mismatch")
 
 // seekTotal retrives the total of all accounts by seeking for each account key.
 func seekTotal(txn *badger.Txn) ([]account, error) {

go.mod

@@ -0,0 +1,15 @@
+module github.com/liquidm/badger
+
+go 1.12
+
+require (
+	github.com/AndreasBriese/bbloom v0.0.0-20190306092124-e2d15f34fcf9
+	github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2
+	github.com/dustin/go-humanize v1.0.0
+	github.com/golang/protobuf v1.3.1
+	github.com/pkg/errors v0.8.1
+	github.com/spf13/cobra v0.0.5
+	github.com/stretchr/testify v1.3.0
+	golang.org/x/net v0.0.0-20190620200207-3b0461eec859
+	golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb
+)