Move to Jest

[eric-burel]

Hi,

Just to open a discussion about improving testing.

Right now, I only managed to setup Mocha to run with Meteor. I struggle to mock Meteor packages, mainly Mongo.

Jest is far more powerful in terms of performance, features, mocking and so on. It does not integrate directly with Meteor though.

This article describes however how to use Jest by mocking Meteor, which is actually quite easy :
https://www.okgrow.com/posts/real-world-unit-testing-with-meteor-and-jest

To work with Vulcan, we would additionally need to rely on specific Webpack loader, exactly the same way we try to do with Storybook. Vulcan code would be fetched from the local install.

Drawbacks:

• no actual integration testing, Meteor is mocked
• need for a 2-repo install, same as for Storybook. However I'd like to use it only for testing Vulcan itself in the first place, so it's ok.
• need to improve the Meteor mock, eg I don't know how to handle Meteor.startup yet. Jest should be configurable enough to allow this though
• not sure what to do about Meteor.isClient, Meteor.isServer, and client/server code more broadly, so basically I am not 100% sure that this is actually doable. Maybe with 2 different configs for client side testing and server side testing, each producing a different build?

Advantages:

• get rid of Mocha
• use Jest
• tests are not pure hell to write anymore
• no need to run/build Meteor to test Vulcan

[cvolant]

Hi,
I am currently studying different options to decide which framework I should adopt.
Vulcan sounds really nice: if it supported Jest and PWA, I would go for it.

[eric-burel]

Vulcan is an active project so even if it may sometimes takes time, it's perpetually improving with new features and usage of the latest libs.

Concerning Jest it's a bit difficult due to the need to build a Meteor app with Webpack. But we managed to do it with Storybook (still under experiment/cleanup) so I am sure it will be ok.

Concerning PWAs, honestly people makes a huge deal about them but technically they are not complex features, basically the hardest part is caching data when offline. Thus I don't think this is the most relevant criteria when choosing a framework. You'll find example of Meteor+Apollo PWA that would adapt in Vulcan.

[SachaG]

Sounds good to me!

not sure what to do about Meteor.isClient, Meteor.isServer, and client/server code more broadly, so basically I am not 100% sure that this is actually doable. Maybe with 2 different configs for client side testing and server side testing, each producing a different build?

I think avoiding Meteor.isClient and Meteor.isServer is a good idea anyway. But I'm not really familiar enough with Jest to know more…

[jimrandomh]

What would be the implications for Vulcan-using projects which already have a significant number of unit tests? Will the frameworks coexist side by side?

[eric-burel]

I think in any case we would need both. So I think Jest tests would have to be isolated in a jest folder, a bit like Storybook, they won't be able to live anywhere in the code. This folder would be ignored by Meteor.
So "meteor test" would run Mocha as usual, and "jest" would run jest tests. We could maintain a legacy behaviour + some specific tests with Mocha, and write new tests with Jest.

[eric-burel]

Gave a first shot at this:

• We need Babel processors instead of the Webpack loader we made for Storybook. This is not a huge deal, we simply need to transform "vulcan-loader" and "scrap-meteor-loader" in a Babel function
• There is also this package that could fit: https://www.npmjs.com/package/babel-plugin-replace-imports. It's basically this, it allow to replace import so you can scrap out "meteor" imports and point "meteor/vulcan:*" to the place you want;
• BUT... you'll be in trouble with isomorphism. That probably means having to differentiate Jest client and server test, and set for them a variable that Jest will be able to read. You might end up needing to name your tests "foobar.client.tests.js" so we can differentiate them
• Instead of altering the main Babel config, we might want to have a Babel config just for Jest

Which sums up as those task:

• Try to write a replacement function with babel-plugin-replace-imports based on the webpack loader we made for Storybook. Edit: actually babel-plugin-transform-rename-import is the correct package for babel 7
• If it fails, write a Babel plugin also based on those loaders
• Create a config for client-side Jest, another for server-side Jest
• Differentiate tests based on their extension, with 2 jest configs
• Use this babel config only in test mode

Update:
Using Babel is probably overkill, Jest moduleNameMapper is just fine for what we want.

Technical subtelty: the npm packages used by your local Vulcan install will be the one from your local Vulcan install. This is NOT the same behaviour as Meteor, which will use your app as the source of truth for NPM packages.
Consequence? If you have annoying issue with core-js, run meteor npm install core-js in the folder that contains Vulcan, not only in your local application.

[juliensl]

This is a nice move ! I can't wait to see the result :D

[eric-burel]

I am almost there, with one slight limitation: it work only if you install node_modules with yarn. I suspect this issue is related to #2680, basically Meteor is implying an older version of Babel/core-js for some reason.

[eric-burel]

It seems that we have to avoid to use "Meteor" and "Mongo" as global variables in Vulcan (which are bad practicies anyway), because it doesn't seem doable to define a global mock from Jest in this case (jest globals can't be functions so they are not sufficient), because the packages are imported and do not belong to users code.
I've raised the question on this SO question related to Meteor and Jest

Edit: this might need a Babel plugin, babel-plugin-transform-globals seems the most appropriate here.

[eric-burel]

Ok I did not succeed with babel-plugin-transform-globals, somehow it isn't applied to the Vulcan install. Best course of action is to remove global "Meteor" and "Mongo" variables from Vulcan and instead import them from "meteor/meteor" explicitely (which is a better practice anyway)

[eric-burel]

Summary of what I've tempted so far

• moduleNameMapper from Jest config allow to replace Meteor packages using regexp

   // Order matters, most specific to the top
      // define mocks here for Vulcan packages
      "^meteor/meteor": "<rootDir>/../.vulcan/mocks/Meteor",
      "^meteor/mongo": "<rootDir>/../.vulcan/mocks/Mongo",
      "^meteor/vulcan:email": path.resolve(
        __dirname,
        "./.vulcan/mocks/vulcan-email"
      ),
       // load vulcan package from your local install
      //"meteor/vulcan:core": `${prefix}core/${defaultPath}`,
      // "meteor/vulcan/some-file"
      "^meteor/vulcan:(.*)/(.*)$": `${vulcanPrefix}$1/$2`,
      // "meteor/vulcan"
      "^meteor/vulcan:(.*)$": `${vulcanPrefix}$1${defaultPath}`,
      // load your own packages
      "^meteor/your-prefix-(.*)/(.*)$": `${yourAppPrefix}$1/$2`,
      "^meteor/your-prefix-(.*)$": `${yourAppPrefix}$1${defaultPath}`,
      // other meteor packages: points to nothing
      "^meteor/(.*)": path.resolve(__dirname, "./.vulcan/mocks/any.js"),
    },

This is equivalent to what we do with Webpack for Storybook using aliases, and to what we can also do with Babel using this plugin. The path to your local Vulcan install + to your app packages are computed using the exact same function.

• Meteor core modules are pointing to local mock (we can't scrap them out like in Storybook, but mocking is better anyway)
• I use 2 configs jest.client and jest.server to differentiate 2 environments. /!\ this is necessary to load Meteor packages correctly, because you have to tell in which environment you want to test. We can play around with the test regex, so that foobar.test.server.js is only run during server test, foobar.test.client.js during client test, and foobar.test.js during both.

  // in jest.server.config.js
  // The glob patterns Jest uses to detect test files
  testMatch: [
    "**/__tests__/**/*.[jt]s?(x)",
    "**/?(*.)+(spec|test).server.[tj]s?(x)",
    "**/?(*.)+(spec|test).[tj]s?(x)",
  ],

Problems

• I did not manage to run Babel plugins on the 2-repo install. I think maybe it uses the Vulcan install Babel config in this case, I don't know yet. In particular, this prevented me to define the Meteor, Mongo and _ globals correctly using relevant Babel plugin. Code below should work but doesn't.

// babel config
const testConfig = {
  // experiment with transformation for global variables
  // Sadly, it doesn't seem to be applied to Meteor modules
  plugins: [
    [
      "transform-globals",
      {
        import: {
          "./.vulcan/mocks/Meteor.js": {
            Meteor: "default",
          },
          "./.vulcan/mocks/Mongo.js": {
            Mongo: "default",
          },
          underscore: {
            _: "default",
          },
        },
      },
    ],
  ], 

• There is a confusing behaviour with the 2-repo install: node_modules from your app are used for your app code, but node_modules from your Vulcan install are used for your Vulcan install too. While in "normal" Meteor, your app node_modules are the single source of truth. So you need to be very careful with your local Vulcan install, eg get core-js 3, the right version of Babel etc.
• Each mock needs additional work, mocking Mongo, functions, collections etc.

So, testing with Jest is an high source of complexity regarding the setup. It is technically doable, but would lead to too many confusion for end users in my opinion. If you have to be a Babel expert to run tests, than you'll have trouble trusting those test.

We may want to stick to Meteor Mocha until there is an official integration of Jest in Meteor or a Babel/Webpack plugin for Meteor code.

Tips to use Mocha more efficiently

• Use MOCHA_GREP=your-test-name to filter out test, like jest your-test-name
• You may need to run  meteor npm install meteor-node-stubs@latest @babel/runtime@latest --save after you ran test (because of this issue objectSpread2 error after running unit test #2680, basically test mode seems to install an older Babel runtime for unknown reasons)
• Use --once to run tests only once (meteor npm run test -- --once)
• Don't forget to set METEOR_PACKAGE_DIRS all the time, it is needed for Meteor run, test, update, etc.
• We use this version of Meteor mocha: https://github.com/meteortesting/meteor-mocha

Possible solution

• Wait for an official integration of Jest in Meteor
• OR wait for an officiel Meteor loader for Webpack or a plugin for Babel that can solve those issues

[haveaguess]

Hi @eric-burel , thanks for this attempt, and sharing your progress. I was just about to start on this journey when I saw your comment in stackoverflow and this backlink. Figured Id see how you were getting on.

Just to understand, is this something we could do with a project that is simpler than what Vulcan is trying to do or is a dead-end even for simpler Meteor apps like ours?

Like you, I'd love to get the benefits of Jest that the author you referred to, seemed to, but don't want to lose time if it's a dead end.

[eric-burel]

Hi ! The problem will be even worse in a normal Meteor application, because the difficulty lies in making Meteor core features available with a Webpack build (or to mock them). It's not really a dead-end, this approach has been successful at least for client-side build, but more a lot of trouble for not much added value if you want also Jest for server-side test.

So it's definitely better to use Mocha solution, until Jest is supported by Meteor (if it ever happens). That means running Jest in Meteor, instead of trying to run Meteor code in Jest+Webpack.