Implement explicit snapshots

See Level/community#118. Depends on
Level/supports#32.

Category: addition

README.md

@@ -387,7 +387,9 @@ console.log(nested.prefixKey('a', 'utf8', true)) // '!nested!a'
 
 **This is an experimental API and not widely supported at the time of writing ([Level/community#118](https://github.com/Level/community/issues/118)).**
 
-Create an explicit [snapshot](#snapshot). Throws a [`LEVEL_NOT_SUPPORTED`](#level_not_supported) error if `db.supports.explicitSnapshots` is not true. For details, see [Reading From Snapshots](#reading-from-snapshots).
+Create an explicit [snapshot](#snapshot). Throws a [`LEVEL_NOT_SUPPORTED`](#level_not_supported) error if `db.supports.explicitSnapshots` is false. For details, see [Reading From Snapshots](#reading-from-snapshots).
+
+Don't forget to call `snapshot.close()` when done.
 
 ### `db.supports`
 
@@ -705,15 +707,25 @@ console.log(foo.path(true)) // ['example', 'nested', 'foo']
 
 ### `snapshot`
 
+#### `snapshot.ref()`
+
+Increment reference count, to register work that should delay closing until `snapshot.unref()` is called an equal amount of times. The promise that will be returned by `snapshot.close()` will not resolve until the reference count returns to 0. This prevents prematurely closing underlying resources while the snapshot is in use.
+
+It is normally not necessary to call `snapshot.ref()` and `snapshot.unref()` because builtin database methods automatically do.
+
+#### `snapshot.unref()`
+
+Decrement reference count, to indicate that the work has finished.
+
 #### `snapshot.close()`
 
 Free up underlying resources. Be sure to call this when the snapshot is no longer needed, because snapshots may cause the database to temporarily pause internal storage optimizations. Returns a promise. Closing the snapshot is an idempotent operation, such that calling `snapshot.close()` more than once is allowed and makes no difference.
 
-After `snapshot.close()` has been called, no further operations are allowed. For example, `db.get(key, { snapshot })` will yield an error with code [`LEVEL_SNAPSHOT_NOT_OPEN`](#level_snapshot_not_open). Any unclosed iterators (that use this snapshot) will be closed by `snapshot.close()` and can then no longer be used.
+After `snapshot.close()` has been called, no further operations are allowed. For example, `db.get(key, { snapshot })` will throw an error with code [`LEVEL_SNAPSHOT_NOT_OPEN`](#level_snapshot_not_open).
 
 #### `snapshot.db`
 
-A reference to the database that created this snapshot.
+A reference to the database that created this snapshot. This refers to the [root database](#subleveldb) when the snapshot was created via a sublevel (which plays no role in snapshots).
 
 ### Encodings
 
@@ -950,10 +962,10 @@ Removing this concern (if necessary) must be done on an application-level. For e
 
 ### Reading From Snapshots
 
-A snapshot is a lightweight "token" that represents the version of a database at a particular point in time. This allows for reading data without seeing subsequent writes made on the database. It comes in two forms:
+A snapshot is a lightweight "token" that represents a version of a database at a particular point in time. This allows for reading data without seeing subsequent writes made on the database. It comes in two forms:
 
 1. Implicit snapshots: created internally by the database and not visible to the outside world.
-2. Explicit snapshots: created with `snapshot = db.snapshot()`. Because it acts as a token, `snapshot` has no methods of its own besides `snapshot.close()`. Instead the snapshot is to be passed to database (or [sublevel](#sublevel)) methods like `db.iterator()`.
+2. Explicit snapshots: created with `snapshot = db.snapshot()`. Because it acts as a token, `snapshot` has no read methods of its own. Instead the snapshot is to be passed to database methods like `db.get()` and `db.iterator()`. This also works on sublevels.
 
 Use explicit snapshots wisely, because their lifetime must be managed manually. Implicit snapshots are typically more convenient and possibly more performant because they can handled natively and have their lifetime limited by the surrounding operation. That said, explicit snapshots can be useful to make multiple read operations that require a shared, consistent view of the data.
 
@@ -1627,7 +1639,7 @@ class ExampleSublevel extends AbstractSublevel {
 
 ### `snapshot = db._snapshot()`
 
-The default `_snapshot()` throws a [`LEVEL_NOT_SUPPORTED`](#errors) error. To implement this method, extend `AbstractSnapshot`, return an instance of this class in an overridden `_snapshot()` method and set `manifest.explicitSnapshots` to `true`:
+The default `_snapshot()` throws a [`LEVEL_NOT_SUPPORTED`](#level_not_supported) error. To implement this method, extend `AbstractSnapshot`, return an instance of this class in an overridden `_snapshot()` method and set `manifest.explicitSnapshots` to `true`:
 
 ```js
 const { AbstractSnapshot } = require('abstract-level')
@@ -1766,7 +1778,7 @@ The first argument to this constructor must be an instance of the relevant `Abst
 
 #### `snapshot._close()`
 
-Free up underlying resources. This method is guaranteed to only be called once. Must return a promise.
+Free up underlying resources. This method is guaranteed to only be called once and will not be called while read operations like `db._get()` are inflight. Must return a promise.
 
 The default `_close()` returns a resolved promise. Overriding is optional.
 

abstract-iterator.js

@@ -18,6 +18,7 @@ const kValues = Symbol('values')
 const kLimit = Symbol('limit')
 const kCount = Symbol('count')
 const kEnded = Symbol('ended')
+const kSnapshot = Symbol('snapshot')
 
 // This class is an internal utility for common functionality between AbstractIterator,
 // AbstractKeyIterator and AbstractValueIterator. It's not exported.
@@ -40,6 +41,7 @@ class CommonIterator {
     this[kLimit] = Number.isInteger(options.limit) && options.limit >= 0 ? options.limit : Infinity
     this[kCount] = 0
     this[kSignal] = options.signal != null ? options.signal : null
+    this[kSnapshot] = options.snapshot != null ? options.snapshot : null
 
     // Ending means reaching the natural end of the data and (unlike closing) that can
     // be reset by seek(), unless the limit was reached.
@@ -363,6 +365,11 @@ const startWork = function (iterator) {
   }
 
   iterator[kWorking] = true
+
+  // Keep snapshot open during operation
+  if (iterator[kSnapshot] !== null) {
+    iterator[kSnapshot].ref()
+  }
 }
 
 const endWork = function (iterator) {
@@ -371,6 +378,11 @@ const endWork = function (iterator) {
   if (iterator[kPendingClose] !== null) {
     iterator[kPendingClose]()
   }
+
+  // Release snapshot
+  if (iterator[kSnapshot] !== null) {
+    iterator[kSnapshot].unref()
+  }
 }
 
 const privateClose = async function (iterator) {

abstract-level.js

@@ -51,12 +51,16 @@ class AbstractLevel extends EventEmitter {
     this[kStatusChange] = null
     this[kStatusLocked] = false
 
+    // Aliased for backwards compatibility
+    const implicitSnapshots = manifest.snapshots !== false &&
+      manifest.implicitSnapshots !== false
+
     this.hooks = new DatabaseHooks()
     this.supports = supports(manifest, {
       deferredOpen: true,
 
       // TODO (next major): add seek
-      snapshots: manifest.snapshots !== false,
+      implicitSnapshots,
       permanence: manifest.permanence !== false,
 
       encodings: manifest.encodings || {},
@@ -323,6 +327,7 @@ class AbstractLevel extends EventEmitter {
     const err = this._checkKey(key)
     if (err) throw err
 
+    const snapshot = options.snapshot != null ? options.snapshot : null
     const keyEncoding = this.keyEncoding(options.keyEncoding)
     const valueEncoding = this.valueEncoding(options.valueEncoding)
     const keyFormat = keyEncoding.format
@@ -335,7 +340,23 @@ class AbstractLevel extends EventEmitter {
     }
 
     const encodedKey = keyEncoding.encode(key)
-    const value = await this._get(this.prefixKey(encodedKey, keyFormat, true), options)
+    const mappedKey = this.prefixKey(encodedKey, keyFormat, true)
+
+    // Keep snapshot open during operation
+    if (snapshot !== null) {
+      snapshot.ref()
+    }
+
+    let value
+
+    try {
+      value = await this._get(mappedKey, options)
+    } finally {
+      // Release snapshot
+      if (snapshot !== null) {
+        snapshot.unref()
+      }
+    }
 
     try {
       return value === undefined ? value : valueEncoding.decode(value)
@@ -368,6 +389,7 @@ class AbstractLevel extends EventEmitter {
       return []
     }
 
+    const snapshot = options.snapshot != null ? options.snapshot : null
     const keyEncoding = this.keyEncoding(options.keyEncoding)
     const valueEncoding = this.valueEncoding(options.valueEncoding)
     const keyFormat = keyEncoding.format
@@ -388,7 +410,21 @@ class AbstractLevel extends EventEmitter {
       mappedKeys[i] = this.prefixKey(keyEncoding.encode(key), keyFormat, true)
     }
 
-    const values = await this._getMany(mappedKeys, options)
+    // Keep snapshot open during operation
+    if (snapshot !== null) {
+      snapshot.ref()
+    }
+
+    let values
+
+    try {
+      values = await this._getMany(mappedKeys, options)
+    } finally {
+      // Release snapshot
+      if (snapshot !== null) {
+        snapshot.unref()
+      }
+    }
 
     try {
       for (let i = 0; i < values.length; i++) {
@@ -809,6 +845,17 @@ class AbstractLevel extends EventEmitter {
     return new DefaultValueIterator(this, options)
   }
 
+  snapshot () {
+    assertOpen(this)
+    return this._snapshot()
+  }
+
+  _snapshot () {
+    throw new ModuleError('Database does not support explicit snapshots', {
+      code: 'LEVEL_NOT_SUPPORTED'
+    })
+  }
+
   defer (fn, options) {
     if (typeof fn !== 'function') {
       throw new TypeError('The first argument must be a function')

abstract-snapshot.js

@@ -0,0 +1,89 @@
+'use strict'
+
+const ModuleError = require('module-error')
+const { noop } = require('./lib/common')
+
+const kStatus = Symbol('status')
+const kReferenceCount = Symbol('referenceCount')
+const kPendingClose = Symbol('pendingClose')
+const kClosePromise = Symbol('closePromise')
+const kRootDatabase = Symbol('rootDatabase')
+
+const STATUS_OPEN = 1
+const STATUS_CLOSING = 2
+
+class AbstractSnapshot {
+  constructor (db) {
+    if (typeof db !== 'object' || db === null) {
+      const hint = db === null ? 'null' : typeof db
+      throw new TypeError(`The first argument must be an abstract-level database, received ${hint}`)
+    }
+
+    // Get root database in case this is a sublevel
+    // TODO: add db property to AbstractLevel too
+    if (db.parent) {
+      db = db.db
+    }
+
+    this[kStatus] = STATUS_OPEN
+    this[kReferenceCount] = 0
+    this[kPendingClose] = null
+    this[kClosePromise] = null
+    this[kRootDatabase] = db
+
+    db.attachResource(this)
+  }
+
+  get db () {
+    return this[kRootDatabase]
+  }
+
+  ref () {
+    if (this[kStatus] !== STATUS_OPEN) {
+      throw new ModuleError('Snapshot is not open: cannot use snapshot after close()', {
+        code: 'LEVEL_SNAPSHOT_NOT_OPEN'
+      })
+    }
+
+    this[kReferenceCount]++
+  }
+
+  unref () {
+    if (--this[kReferenceCount] === 0 && this[kPendingClose] !== null) {
+      this[kPendingClose]()
+    }
+  }
+
+  async close () {
+    if (this[kClosePromise] !== null) {
+      // First caller of close() is responsible for error
+      return this[kClosePromise].catch(noop)
+    }
+
+    // Wrap to avoid race issues on recursive calls
+    this[kClosePromise] = new Promise((resolve, reject) => {
+      this[kPendingClose] = () => {
+        this[kPendingClose] = null
+        privateClose(this).then(resolve, reject)
+      }
+    })
+
+    // If working we'll delay closing, but still handle the close error (if any) here
+    if (this[kReferenceCount] === 0) {
+      this[kPendingClose]()
+    }
+
+    return this[kClosePromise]
+  }
+
+  async _close () {}
+}
+
+const privateClose = async function (snapshot) {
+  // There's no need for a closed status atm (after _close)
+  snapshot[kStatus] = STATUS_CLOSING
+  await snapshot._close()
+  snapshot[kRootDatabase].detachResource(snapshot)
+}
+
+exports.AbstractSnapshot = AbstractSnapshot

index.d.ts

@@ -38,4 +38,8 @@ export {
   AbstractSublevelOptions
 } from './types/abstract-sublevel'
 
+export {
+  AbstractSnapshot
+} from './types/abstract-snapshot'
+
 export * as Transcoder from 'level-transcoder'

index.js

@@ -6,3 +6,4 @@ exports.AbstractIterator = require('./abstract-iterator').AbstractIterator
 exports.AbstractKeyIterator = require('./abstract-iterator').AbstractKeyIterator
 exports.AbstractValueIterator = require('./abstract-iterator').AbstractValueIterator
 exports.AbstractChainedBatch = require('./abstract-chained-batch').AbstractChainedBatch
+exports.AbstractSnapshot = require('./abstract-snapshot').AbstractSnapshot

package.json

@@ -16,6 +16,7 @@
     "abstract-chained-batch.js",
     "abstract-iterator.js",
     "abstract-level.js",
+    "abstract-snapshot.js",
     "index.js",
     "index.d.ts",
     "lib",

types/abstract-level.d.ts

@@ -3,6 +3,7 @@ import * as Transcoder from 'level-transcoder'
 import { EventEmitter } from 'events'
 import { AbstractChainedBatch } from './abstract-chained-batch'
 import { AbstractSublevel, AbstractSublevelOptions } from './abstract-sublevel'
+import { AbstractSnapshot } from './abstract-snapshot'
 
 import {
   AbstractIterator,
@@ -235,6 +236,14 @@ declare class AbstractLevel<TFormat, KDefault = string, VDefault = string>
    */
   valueEncoding (): Transcoder.Encoding<VDefault, TFormat, VDefault>
 
+  /**
+   * Create an explicit snapshot. Throws a `LEVEL_NOT_SUPPORTED` error if
+   * `db.supports.explicitSnapshots` is false.
+   *
+   * Don't forget to call `snapshot.close()` when done.
+   */
+  snapshot (): AbstractSnapshot<typeof this>
+
   /**
    * Call the function {@link fn} at a later time when {@link status} changes to
    * `'open'` or `'closed'`. Known as a _deferred operation_.

types/abstract-snapshot.d.ts

@@ -0,0 +1,30 @@
+/**
+ * A lightweight token that represents a version of a database at a particular point in
+ * time.
+ *
+ * @template TDatabase Type of the database that created this snapshot.
+ */
+export class AbstractSnapshot<TDatabase> {
+  /**
+   * A reference to the database that created this snapshot.
+   */
+  db: TDatabase
+
+  /**
+   * Increment reference count, to register work that should delay closing until
+   * {@link unref} is called an equal amount of times. The promise that will be returned
+   * by {@link close} will not resolve until the reference count returns to 0. This
+   * prevents prematurely closing underlying resources while the snapshot is in use.
+   */
+  ref (): void
+
+  /**
+   * Decrement reference count, to indicate that the work has finished.
+   */
+  unref (): void
+
+  /**
+   * Close the snapshot.
+   */
+  close (): Promise<void>
+}