How to make Mongoose hydration faster? 📈

[rubenvereecken]

We've got a quickly growing project so we're reevaluating some assumptions that we made when we started the project. One of the reasons is that performance is becoming a concern.

We already knew Mongoose was faster if we didn't hydrate results (using query.lean()), but we heavily underestimated how much faster. A simple benchmark where we fetch a 36kb document (containing subdocuments), then serialise it to json (30 iterations in each case):

• lean: 18ms
• hydrated: 41ms

Granted, it's a big document, but over 20ms on an application server is a massive performance price to pay. This CPU spend was also validated using tracing — it all happens in the Mongoose layer, the database call is the same.

We're relying hard on hydrated documents because we have some nice transformation logic and quite a lot of methods that require hydrated documents. Though, not really. Really what we need are lean documents with a thin layer of logic on top — something we thought Mongoose provided. There's some good news: there's a way out for us. Using lean + mongoose-lean-methods got us some of the way there (no noticeable performance impact), but it's a big undertaking.

Finally, the question, an invitation for discussion: where is the majority of this time spent? It's completely blocking so it's nothing IO-related. Would there be a way for us to limit this expensive computation? Is it really necessary? Even thinking of all the goodies Mongoose offers and having a skim of document.js, I just can't think of where 20ms might be spent.

Ideally what we have is all of the simple stuff Mongoose offers: transformations, methods, virtuals etc, which are all very cheap, and ignore whatever turns out to be so expensive.

[vkarpov15]

We're actively working on some issues related to this. See #11541, #10400, etc. Minimizing performance overhead and memory usage has been our top priority over the last year or so. We've made some good progress, but we still have a ways to go.

If you're willing to provide us the code of the benchmark you ran, that would be helpful.

[rubenvereecken]

@vkarpov15 I thought of how to give you access to our benchmark but it's really in the data, rather than any code. The code is a simple difference between a lean query and a non-lean query. The data looks like this (abstracted):

class DocSchema {
  // simple primitives and simple subdocuments, totalling 10kb
  data: ...
  // array of subdocs, often 30 or so, totalling 30kb
  subdocs: ...
}

---

The way we ended up going: We use a version of mongoose-lean-methods plus mongoose-lean-virtuals and mongoose-lean-defaults. This has barely any performance impact and gets us some nice goodies. What was missing was transforms, especially recursive transforms (automatically transforming all subdocuments as well).

We worked around this by having all of our classes implement CommonSchema, which has a method toJSON which also calls toJSON on all of its subdocuments. Any class can override this for custom behaviour (such as depopulating or removing fields).

End result: we're converting as many queries to lean as possible without having to give up anything major, as long as the documents aren't being saved or anything like that.

Appreciate your hard work on this @vkarpov15 !