Write Tests For Server Code

Before, during, or after making your changes to the codebase, you should spend some time considering how to test your code to ensure correct behavior; you must test your code to ensure correct behavior.

Sometimes, existing tests will cover everything you need to test, but sometimes you will need to write new tests.

If you're changing existing user-facing functionality, or covering newly discovered edge cases, you should add new JavaScript tests. If you're changing the internal mechanics of a feature, you should add a new C++ unit test.

Adding a New C++ Unit Test

A new-style C++ unit test file contains uses of the TEST(case_name, test_name) and TEST_F(fixture_name, test_name) macros. The TEST macro constructs a single test class, containing whichever setup and asserts you provide inside the function. TEST_F, on the other hand, constructs a test class which inherits from the base class (or "fixture") fixture_name. Thus, one example use of TEST_F might be:

class FixtureClass : public mongo::unittest::Test {
protected:
    int myVar;
    void setUp() { myVar = 10; }
};

TEST_F(FixtureClass, TestThatUsesFixture) {
    ASSERT_EQUALS(10, myVar);
}

Refer to unittest/unittest.h for a full record of all available assertions and other unit test resources.

After writing a new C++ unit test, the next step is to declare the test as a SCons target in an SConscript file. This is easiest to show by example:

There are multiple uses of the TEST macro in the unit test parsed_projection_test.cpp found in the src/mongo/db/query subdirectory. To build parsed_projection_test.cpp, it must be declared inside an SConscript file found in the same src/mongo/db/query subdirectory.

A SConscript declaration has three parts: a target name, a source referring to the test file, and a list of LIBDEPS or library dependencies.

env.CppUnitTest(
    target="parsed_projection_test",
    source=[
        "parsed_projection_test.cpp"
    ],
    LIBDEPS=[
        "query_planner",
    ],
)

In this case, the libdep "query_planner" is declared inside the same SConscript file. If, instead, there is a libdep of the form $BUILD_DIR/mongo/bson or similar, that library is declared elsewhere inside the project. For example, the bson library declaration can be found in src/mongo/SConscript.

Each library declaration includes a target (the name of the library), a list of source files, and libdeps of its own. Put simply, libraries provide a method of organizing and partitioning the code base. One result is that unit tests do not require a full recompilation to be used.

An example that contains uses of the TEST_F macro is the unit test collection_metadata_test.cpp in the src/mongo/s directory. This unit test is declared inside the src/mongo/s subdirectory's SConscript file.

Adding a New JavaScript Test

JavaScript tests are written as a series of shell operations. If you are changing any user-facing functionality, add a new JS test or edit an existing test to confirm the correctness of your change.

Under the jstests directory, subdirectories exist with names that categorize the tests found within the directories. The jstests/core subdirectory is for tests that are valid against any kind of mongodb server (i.e. mongos/mongod/a replica set). The core tests runs against those different environments. When these tests are run, there will be a database set up for their use by the test suite.

If your test is for basic mongodb behavior, place the test in jstests/core. Tests that are appropriate for replica sets only go in jstests/replsets directory, and tests for sharded clusters go in jstests/sharding. You will need to build your own replica set or sharded cluster in the respective directories.

If your tests do not fall into one of these categories, one of the other subdirectories may be more appropriate for your tests.

In the process of testing-the-mongodb-server, you may have looked at the list of Smoke.py suites. These suites correspond to the core tests being run with a number of different backend db configurations. For example, with authorization, with a small oplog, against a mongos, in parallel with each other, in a mode that downconverts writes, etc.

Assertions 101

When writing your JavaScript tests, use assertions to validate your tests assumptions and invariants. These are mainly defined in src/mongo/shell/assert.js, though they can be found elsewhere in the repository as well. Some examples of assertions and their usage:

Assertion	Usage
assert(expression, message)	Asserts that expression evaluates to a true-like value. If not, prints a stack trace and message.
assert.eq(a, b, message)	Asserts that two values are equal (tested with == and friendlyEqual)
assert.contains(item, object, message)	Asserts that item is a value in the Object (or Array).

Use the message argument in assert methods to say something about what the assertion failure means.

General JS Test Tips

• Do not use db as a variable name in JS tests. The mongo shell defines a db object, and conflicts with your variable can be hard to diagnose.
• Do not introduce non-deterministic behavior into a test. We do not want random test failures! Be wary when testing against log output or features with distributed elements, like replSet elections.
• Do not assume that two things will happen immediately after each other. If you are using timeouts, reconsider their usage -- and even if you are 100% sure you need them, think again.

The Problem of Parallelization

Server tests must be able to run in parallel and still succeed. Here are some tips to make satisfying this constraint easier:

• Try to use a unique collection name, for example one named after the test:

  coll = db.jstests_currentop
  

• If checking on current operations, make sure to add an ns filter.

  db.currentOp( { ns: "test.mytest" } )
  

• If possible, try to avoid things that are global to mongod or the database (oplog, profiling, fsync)

If the test cannot be run in parallel, add it to the blacklist in skipTests in jstests/libs/parallelTester.js.

A Note on Write Commands

Prior to the 2.6 release, writes (inserts, updates, and removes) were sent over the wire protocol to a MongoDB host as a specially formatted message type. Those old write messages do not allow a response, and so to detect errors JS tests must later invoke the "getLastError" (GLE) command on the same connection/socket. GLE is still supported for backwards compatibility, and the compatibility suite tests this old framework.

Starting in 2.6, we implemented write commands. Write operations (inserts, updates, and removes) will return a WriteResult object, which contains any error that occurred during the write. JS tests often consist of checking the contents of these WriteResult objects.

For example, the following operation attempts to insert a document:

db.coll.insert({ _id : 0, hello : "world" })

Upon successful operation, the method returns the following object:

WriteResult({ "nInserted" : 1 })

If another operation attempts to insert a document with a duplicate _id:

db.coll.insert({ _id : 0, hello : "world" })

The method errors and returns the following object which includes information on the error:

WriteResult({
  "nInserted" : 0,
  "writeError" : {
    "code" : 11000,
    "errmsg" : "insertDocument :: caused by :: 11000 E11000 duplicate key error index: test.coll.$_id_  dup key: { : 0.0 }"
  }
})

Connection errors and other basic errors that occur before the write (such as authentication), as well as interrupted operations, are thrown as JavaScript exceptions.

Spawning New Instances Inside a JS Test

The mongo shell provides the following three utilities for spawning mongod standalone instances, replica sets, and sharded clusters:

• MongoRunner,
• ReplSetTest, and
• ShardingTest.

These tools spawn processes that run in the background of your JavaScript test but will block your test script until the mongod instance (or replica set, or sharded cluster) is ready to accept connections. You may then connect to a mongos or a mongod from your script.

These tools obviate any need to manually start and stop potentially complicated setups before running tests. These utilities also make re-creating specific situations with replica sets and sharded clusters easy to duplicate.

Similarly, ToolTest allows you to test MongoDB tools, such as mongodump or mongorestore, from the shell.

This section will cover the basics of each utility, but for more specific information, refer to the source code for each utility.

A great, well-commented example of a JS test using ReplSetTest is replset1.js. The name of the test however is an example of bad test naming; make your test names are meaningful.

Similarly, cleanup_orphaned_cmd_hashed.js is a good example of a JS test using ShardingTest.

For an example of a test that uses ToolTest, see dumprestore2.js.

Introduction to MongoRunner

MongoRunner_ is a static object which manages running processes. It has five core functions:

• MongoRunner.runMongod(...)
• MongoRunner.stopMongod(...)
• MongoRunner.runMongos(...)
• MongoRunner.stopMongos(...)
• MongoRunner.runMongoTool(...)

Each of these functions takes a parameter object. All parameter fields, except certain special names, are mapped to command line options for the mongod, mongos, and mongo tool objects that are executed. The real utility of the MongoRunner class is the ability to remember and restart mongod and mongos processes with the same parameters as were initially used.

var conn = MongoRunner.runMongod({ smallfiles : "", oplogSize : 40, noprealloc : "" });
MongoRunner.stopMongod(conn);
conn = MongoRunner.runMongod({ restart : conn });

Introduction to ReplSetTest

The ReplSetTest_ object wraps MongoRunner to provide an easy way to spin up a replica set. A good replica set test to use as reference is replset1.js_.

ReplSetTest takes two basic parameters: the name of the set and the number of nodes to create. The third parameter, nodeOptions, is passed to the MongoRunner.runMongod call which starts the nodes, allowing arbitary command-line options to be set. Unless you're running more than one ReplSetTest at once, it's usually simpler to omit the "name" field and rely on the defaults.

var rst = new ReplSetTest({ nodes: 3, nodeOptions : {...} });

note

Using rs as the variable name is not recommended, since doing so will mask the replica set utility functions available globally under rs.<>.

When the constructor returns, the set will not be running and will not be initialized. You must perform both steps yourself using:

rst.startSet({...});
rst.initiate();

The rst.startSet() function returns after all node mongod processes have been started and are connectable. Options passed here are sent to the MongoRunner when starting the nodes. The rst.initiate() operation returns immediately, but there is no guarantee that the replica set at this point has any nodes as primary or secondary.

In order to get a connection to the replica set's primary node, we can run:

var primary = rst.getPrimary();

The primary node returned here will be a Mongo connection object. Note that the above function waits for the primary node to be elected - if a majority of replica set members are unavailable, this function will timeout and error. In contrast, we explicitly must wait for the rest of nodes to reach secondary state:

rst.awaitSecondaryNodes();
var secondaries = rst.getSecondaries();

You can access nodes as follows:

var node = rst.nodes[0];
rst.waitForState(node, ReplicaSetTest.State.PRIMARY|SECONDARY|RECOVERING|ARBITER);

When you have finished with the replica set, stop the replica set with rstConn.stopSet().

Introduction to ShardingTest

The ShardingTest_ object builds upon the ReplSetTest and MongoRunner abstractions to allow the creation of a sharded cluster on the local machine with default ports and data directories. A good example of a sharding test to use as reference is cleanup_orphaned_cmd.js_.

var st = new ShardingTest({
  shards : 0|1|2|3..., // the number of shards to create
  mongos : 0|1|2|3..., // the number of mongoses to create
  verbose: 0|1|2..., // the level of verbosity
  other : {
    mongosOptions : {...}, // options passed to the mongos processes
    configOptions : {...}, // options passed to the config server processes
    separateConfig : true|false, // whether to reuse the first shard as a config server
    sync : true|false, // whether we want 3 config servers or 1
    rs : true|false, // whether or not to create replica set shards
    rsOptions : {...} // options passed to the replica set objects (if replica set shards)
    shardOptions : {...}, // options passed to the shard processes (if not replica set shards)
  }
});

The example above specifies more options than are commonly used in a given test: in many cases, only the shards and mongos options are necessary. More options are also available -- refer to the source code for a full accounting.

You can access connections to the initialized cluster using the following properties:

st.s0|1|2|3... // Connections to the mongos processes of the cluster, also available at st._mongos
st.config0|1|2... // Connections to the config processes of the cluster, also available at st._configServers
st.shard0|1|2... // Connections to (non-replica-set) shards of the cluster, also available at st._connections
st.rs0|1|2... // ReplSetTest objects of replica set shards in the cluster, also available at st._rsObjects

There are also some specific functions to be called against the ShardingTest:

st.printShardingStatus(); // Prints status of cluster
st.stopBalancer(); // Stops and waits for balancer to stop
st.startBalancer(); // Starts and waits for balancer to start

Remember to stop the cluster when you are finished with it using st.stop().

Multiversion Testing with MongoRunner

MongoRunner can be used to write multiversion tests using a special binVersion parameter. A good example of a multiversion test on which to base your own tests is 2_test_launching_cluster.js.

var mongod24 = MongoRunner.runMongod({ binVersion : "2.4" });

Under the covers, the binVersion parameter simply indicates to MongoRunner that it should replace the normal executable name of "mongod" with the new name "mongod-2.4". Assuming the appropriate executables exist on the system running multi-version tests, mongod24 will be a connection to a mongod process running some "2.4.x" version of mongodb.

However, all of this multi-version testing fundamentally relies on simple symlinks or executables with particular installed names. The abstractions themselves know nothing of versions. You will need to create symlinks in your main Mongo folder to the relevant mongo/mongod/mongos files, for example:

cd  ~/mongo
ln -s ~/mongo-2.4.6/bin/mongo mongo-2.4

If you're working on a Linux machine, you can make use of a prewritten script to setup these symlinks, found in buildscripts/setup_multiversion_mongodb.py.

Because the ReplSetTest and ShardingTest objects wrap calls to MongoRunner, and allow the passing of arbitrary parameters to all parts of a replicated or sharded system, it is simple to start an entire cluster at whatever version is required. For example:

var rst = new ReplSetTest({ nodes : 3 });
rst.startSet({ binVersion : "2.4" });

Alternatively, you may require a mixed-version cluster or replica set. To facilitate this kind of startup, MongoRunner provides a versionIterator function, which returns an object that iterates one-by-one through the version options. As an example:

var version = MongoRunner.versionIterator(["2.0", "2.2"]);
var options = {
  mongosOptions : { binVersion : version },
  configOptions : { binVersion : version },
  shardOptions : { binVersion : version },
  [rsOptions : { binVersion : version },]
  separateConfig : true,
  sync : true,
  [rs : true]
};
var st = new ShardingTest({ shards: 2, mongos: 2, other: options });

The resulting cluster will be composed of 1/2 v2.0 mongodb processes and 1/2 v2.2 mongodb processes. Upgrading is just a matter of restarting mongod/s processes at a different binVersion.

note

Warning: The versionIterator works randomly, which means that you cannot rely on the "first" process corresponding to the "first" version you passed in.