Storage Plugin

Create the Storage Plugin

In this section we create the skeleton of our storage plugin:

• The Maven module
• The Plugin Config
• The Plugin Class

Create the Module

Choose a name for the Maven module that will hold your storage plugin. Let's call ours "example". Storage plugins typically go in the contrib module, in a subdirectory named storage-example.

To create the project in Eclipse:

• Select the drill-contrib-parent node in the Eclipse Package Explorer.
• Choose Maven → New Maven Module Project from the context menu.
• Give your project the storage-example name.
• Accept the other defaults and create the project.
• Edit the resulting pom.xml file to add your needed dependencies. Use other storage plugins as a "cheat sheet" to see what is needed.
• Edit drill-contrib-parent/pom.xml to verify your module appears in the module list. Add it if missing:

  <modules>
    ...
    <module>storage-example</module>
    <module>data</module>
    ...
  </modules>

• (Review this part.) Get Eclipse to know about your project. Use File → Import → Existing Maven Projects. Select the Drill root. Your project should now appear in the Package Explorer as storage-example. (If this does not work, try selecting drill and Maven → Update Project.
• Eclipse named your new package org.storage.example. Rename this package to org.apache.drill.exec.store.example.
• Do the same for the test package.
• Run the Maven-provided 'App' class to ensure everything works. Select the class name, then Debug as → Java Application from the context menu.
• Delete the example App and AppTest classes.

Storage Plugin Config Class

The config class provides all configuration information needed to execute your plugin, except the query-specific information.

• Create the ExampleStoragePluginConfig class:

@JsonTypeName(ExampleStoragePluginConfig.NAME)
public class ExampleStoragePluginConfig extends StoragePluginConfigBase {

  public static final String NAME = "example";

  public ExampleStoragePluginConfig() {
  }

  @Override
  public boolean equals(Object o) {
     if (o == this) {
      return true;
    }
    if (o == null || !(o instanceof ExampleStoragePluginConfig)) {
      return false;
    }
   return Objects.equal(field1 field2);
  }

  @Override
  public int hashCode() {
    return Objects.hashCode(field1, field2);
  }

  @Override
  public String toString() {
    return getClass().getSimpleName() +
      " [field1=" + field1 +
      ", field2=" + field2 + "]";
  }
}

Add the config fields needed by your plugin. You will need:

• The field itself, which must be public and should be a simple Java type such as an int or String. (More complex types are possible, but can be tricky. Look at other config classes for examples.)
• Add either a public setFoo method or add the field to the constructor.
• Create a public getFoo method for each field.
• Add the field to the hashCode, equals and toString methods.

Storage Plugin Class

The storage plugin class is Drill's main entry point to your plugin.

• Create the storage plugin class in Eclipse. The initial file should look like this:

public class SumoStoragePlugin extends AbstractStoragePlugin {
  private final ExampleStoragePluginConfig config;

  public ExampleStoragePlugin(ExampleStoragePluginConfig config,
      DrillbitContext context, String name) throws IOException {
    super(context, name);
    this.config = config;
  }

  @Override
  public StoragePluginConfig getConfig() { return config; }

  @Override
  public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus parent)
      throws IOException {
    // TODO Auto-generated method stub
  }

  @Override
  public boolean supportsRead() { return true; }
}

The constructor has special meaning to Drill: Drill uses the type of the first argument (here ExampleStoragePluginConfig ) to match a storage plugin class with a storage plugin config class.

The supportsRead() method says that our plugin can read data (some can only support writes.)

We'll fill in the others as we go.

Drill Config File

Create the plugin's config file:

• Under your src folder, create a resources subfolder.
• Select the resources folder in the Package Explorer, then, from the context menu: Build Path → Use as Source Folder.
• Under resources, create drill-module.conf (easiest to just copy from another storage plugin such as OpenTSDB):

drill.classpath.scanning: {
  packages += "org.apache.drill.exec.store.example"
}

The above tells Drill about your storage plugin.

Bootstrap Config

Optionally create the default config, if any, you want for a newly-installed Drill. These defaults are also available to unit tests.

• Create the file bootstrap-storage-plugins.json in your resources folder.
• Put into the file the JSON-encoded version of your default configuration.

Or, if there is no good default, just omit this file. In this case, users will have to how to enter the JSON by hand in the Drill web console.

First Test

Create a test for your plugin:

public class TestExamplePlugin extends ClusterTest {

  @BeforeClass
  public static void setup() throws Exception {
    ClusterFixtureBuilder builder = new ClusterFixtureBuilder(dirTestWatcher);
    startCluster(builder);

    StoragePluginRegistry pluginRegistry = cluster.drillbit().getContext().getStorage();
    ExampleStoragePluginConfig config =
        new ExampleStoragePluginConfig(/* Your fields here */);
    config.setEnabled(true);
    pluginRegistry.createOrUpdate(ExampleStoragePluginConfig.NAME, config, true);
  }

  @Test
  public void test() {
    fail("Not yet implemented");
  }
}

The setup() method does some "magic" to start an in-process Drill "cluster" (really, one Drillbit), create an instance of your plugin config, and register that config in Drill.

Notice that the test case does nothing yet.

To test, set a breakpoint in the constructor of your storage plugin. Run your test. If Eclipse hits the breakpoint, then you've got everything wired up correctly. If not, go back and review the above steps:

• The config class is marked as Jackson serializable.
• The storage plugin takes the config as its first option as shown above.
• The drill-module.conf file exists and adds the correct package to the class path.
• The test code is correct.