Feature request: Support FlexSearch 0.7.x

[haysclark]

A new major version of FlexSearch 0.7.0 has been released that might have some breaking changes.

[haysclark]

My first thought was, "Move 'flexsearch' to peerDependencies". Unfortunately, this would result in users having to install both gatsby-plugin-local-search and flexsearch even if they are NOT using flexsearch for local search.

Maybe it would be worth the effort to upgrade the library to be more modular and have the 'engines' broken out into sub-plugin? Sorta like the structure of the gatsby-transformer-remark ecosystem.

gatsby-transformer-remark's child plug-in loading logic

proposed gatsby-config.js:

module.exports = {
  plugins: [
    {
      resolve: "gatsby-plugin-local-search",
      options: {
        // engine: "flexsearch", <-- omit and rely on plugins to define engine?
        name: "pages",
        // new section
        plugins: [
          `gatsby-local-search-flexsearch`,
          // OR
          {
            resolve: `gatsby-local-search-flexsearch`,
            options: {
              // ... 
            },
          },
        ],
        // end of new section
        query: `
  query {
    allMarkdownRemark(sort: { fields: [frontmatter___date], order: ASC }) {
      nodes {
        id
        fields {
          slug
        }
        frontmatter {
          date(formatString: "MMMM DD, YYYY")
          title
        }
        rawMarkdownBody
      }
    }
  }
`,
        ref: "id",
        index: ["title", "body"],
        store: ["id", "title", "body", "date", "slug", "type"],
        normalizer: ({ data }) =>
          data.allMarkdownRemark.nodes.map(node => ({
           // ... code
          })),
      },
    },
  ],
}

[angeloashmore]

Hey @haysclark, this is a great recommendation. Splitting into sub-plugins also opens the system up to others search engines more easily.

The plugins option is required to be an array, like you have in your example. In the case of this plugin, multiple sub-plugins isn't applicable since it should only produce one index. Is there any need to build a FlexSearch and Lunr index for the same set of data?

Rather than use plugins we could use the engine option as a way to load the sub-plugin. For example, the config could look something like this:

{
  resolve: 'gatsby-plugin-local-search',
  options: {
    name: 'pages',
    engine: 'gatsby-local-search-flexsearch',
    query: `
      query {
        allMarkdownRemark(sort: { fields: [frontmatter___date], order: ASC }) {
          nodes {
            id
            fields {
              slug
            }
            frontmatter {
              date(formatString: "MMMM DD, YYYY")
              title
            }
            rawMarkdownBody
          }
        }
      }
    `,
    index: ['title', 'body'],
    store: ['id', 'title', 'body', 'date', 'slug', 'type'],
    normalizer: ({ data }) =>
      data.allMarkdownRemark.nodes.map((node) => ({
        // ... code
      }))
  },
}

This keeps the API similar to the existing API and allows us to migrate from engine: 'flexsearch' to engine: 'gatsby-local-search-flexsearch' progressively. For example, we could detect the current "flexsearch" and "lunr" options and warn the user to install the plugin and change the config.

gatsby-local-search-flexsearch would be loaded dynamically with its own FlexSearch-specific logic.

What do you think?

[haysclark]

@angeloashmore Awesome! Yeah that is a good point about 1:1 relationship as well at the point about backwards compatibility. Just to clarify, are you imagining we can support providing custom options to the child engines?

e.g. (this is 99% what you posted, just added the options API for clarity.)

{
  resolve: 'gatsby-plugin-local-search',
  options: {
    name: 'pages',
    // for legacy and when providing no custom options
    engine: 'gatsby-local-search-flexsearch',
    // for providing custom options
    engine: {
      resolve: `gatsby-local-search-flexsearch`,
      options: {
        // ... 
      },
    },
    query: `
      query {
        allMarkdownRemark(sort: { fields: [frontmatter___date], order: ASC }) {
          nodes {
            id
            fields {
              slug
            }
            frontmatter {
              date(formatString: "MMMM DD, YYYY")
              title
            }
            rawMarkdownBody
          }
        }
      }
    `,
    index: ['title', 'body'],
    store: ['id', 'title', 'body', 'date', 'slug', 'type'],
    normalizer: ({ data }) =>
      data.allMarkdownRemark.nodes.map((node) => ({
        // ... code
      }))
  },
}

[angeloashmore]

Good catch on the engine options. I forgot about that part!

The current API allows for an engineOptions value. Currently, it is only provided to the FlexSearch engine and is effectively passed through to the index creator as is. In the approach we're discussing, it would be passed to the engine plugin directly for the plugin to do what it needs with it.

For the sake of API simplicity, I'd lean toward using the separate engine and engineOptions properties rather than a single engine option that takes a string or an object with nested properties. I know your example mimics the plugins API, but in this case, I think it may introduce added complexity unnecessarily.

Do you think there would be a benefit to using the resolve/options form instead?

[haysclark]

Do you think there would be a benefit to using the resolve/options form instead?

I don't think it is really required, I was just following the Gatsby conventions that I'm used too seeing. I like the nested structure of the resolve/options format.

[haysclark]

@angeloashmore Hmm.... it looks like FlexSearch 0.7.x's import/export API's have changed quite a bit. FlexSearch 0.7.x exports to multiple data blobs that have to be imported in separate chunks.

Additionally, there appears to be a flaw in the new export() API, because one can not determine when the exporting process has been completed. nextapps-de/flexsearch#235

[gabrielcsapo]

is there still work being done adding 0.7 support?

[angeloashmore]

@gabrielcsapo There isn't any active development for 0.7 support, but I would be happy to review a PR. To keep the upgrade simple, we can either go the plugin route described in this issue, or we can keep the existing system while supporting 0.7.

[davidpaulsson]

Hey everyone :) Does anyone have a working solution for Gatsby + Flexsearch@7+?