Device specific simulation?

[adamaji]

Hi folks,

I'm interested in building some simulator classes for specific Plus devices. I'm wondering if this is something that could belong in the repo, or if this is even the correct approach to what I'd like to do.

Basically, our group has a robotic setup with multiple devices (probes and trackers) that are controlled by a separate application that interfaces with a PlusServer. I'd like to automate testing of this external application without needing actual probes by simulating the hardware that is run by the application.

I understand that there is a UsSimulatorVideoSource class, but I'd like to combine this functionality of simulating the Us stream with faking certain functions to more closely simulate the state of the device (e.g., calling fakeDevice->freezeProbe() freezes the simulated stream, or calling fakeDevice->setVoltage() sets some internal variable which maintains that voltage state). Ideally, I'd like to simulate our entire robotic setup in a scene in Slicer (similar to how the example scene is set up for the UsSimulator demo) and have that scene interact with our application as if it were just connecting to physical devices.

So far, I have been hacking together simulated actions by rerouting commands sent to the PlusServer, but this is both tedious and feels like a bad way to design a testing framework. What I'd like to do is to be able to instantiate the real and fake device in the same way, and call the same methods from each. This would mean that the devices I'm interested in simulating would need an interface which both the real and fake device classes would inherit from. So something like

class MyDeviceInterface
{
  public:
    virtual PlusStatus freezeProbe();
}

class MyRealDevice : public MyDeviceInterface
{
  public:
    virtual PlusStatus freezeProbe()
    {
      // actually freeze a real probe stream
    }
}

class MyFakeDevice : public MyDeviceInterface
{
  public:
    virtual PlusStatus freezeProbe()
    {
      // freeze a simulated probe stream
    }
}

So in the application I can do something like

MyDeviceInterface* device;
if (is_simulation)
{
  device = new MyFakeDevice();
}
else
{
  device = new MyRealDevice();
}
device->freezeProbe();
delete device;

Does this approach seem viable and appropriate? Happy to clarify things if need be.

cc @dzenanz @jamesobutler

[dzenanz]

This sounds good to me. I suppose that MyDeviceInterface should inherit from vtkPlusUsDevice, and the existing class redirected to inherit from MyDeviceInterface instead of directly from vtkPlusUsDevice.

Side note: docs state that "Refraction, speed of sound, and beamwidth are not modeled.", but SpatialModels in the example config file have SoundVelocityMPerSec property. Are the docs out of date, or is the config file forward looking?

[adamrankin]

This class does something similar:

https://github.com/PlusToolkit/PlusLib/blob/master/src/PlusDataCollection/FakeTracking/vtkPlusFakeTracker.h

So yes, you would create a new device type and in your if/else you just create the appropriate device.

[adamaji]

I suppose as it relates to the PlusLib repository, instead of creating an entirely new device that only inherits from vtkPlusDevice, would it be appropriate to create an abstract class / interface which inherits from vtkPlusDevice and is inherited by both the real and simulated devices like Dzenan is suggesting above?

[lassoan]

The ultrasound simulator device should be extensible with whatever you need (unless you want to add something very special that could ever happen in your ultrasound system). For example, adding freeze function to the simulator would be useful in general.

You can simulate trackers and other sensors in Plus, too, but it is probably simpler to do it using Python scripting in Slicer. You can add an observer to the transforms that the user can change and update all the derived transforms (if the transform tree does not take care of something automatically). If you want to simulate independent, non-user initiated actions (e.g., patient breathing motion) then you can add a QTimer that calls a function N times per second to update certain transforms.

[adamaji]

Yes, I think there are cases where I would like to use simulated functions that are specific to only a small subset of devices, e.g., if I wanted to call SetWobbleRate for a wobbling transducer this probably doesn't belong in UsSimulatorVideoSource. Something like FreezeProbe I agree would be useful in the existing simulator class, so perhaps the hierarchy could look like the following?

  vtkPlusDevice
        |
  UsSimulatorVideoSource  (includes FreezeProbe)
        |
   MyAbstractWobblingDevice  (includes virtual SetWobbleRate)
      /                  \
MyFakeWobblingDevice   MyRealWobblingDevice

[jamesobutler]

My understanding is that Adam is essentially wanting to test the code from an UltrasoundDevice which essentially has all the device API calls removed so that is purely Plus code. Then it could be used to confirm if appropriate device internal properties are being set correctly. Of course testing if the image data changes wouldn't be accomplished here as you would have to confirm with hardware that your device actually did what you asked it to do.

[adamaji]

Yes, something to that effect. I think ultimately the goal is to have something that looks like it acts like the real device from the outside -- the internals perhaps don't matter as much, except for the main Plus guts (frame acquisition/streaming/saving to disk). And yes real image data changes could probably not be tested, but I think actions like freezing the stream and such could work. I'm not sure to what extent the simulated streams can account for the parameters that we'd like to change, but it would be nice if we could see that changing a parameter changes the simulated stream.

[jamesobutler]

it would be nice if we could see that changing a parameter changes the simulated stream.

This seems nice if you were trying to provide a demo at tradeshow for instance, but I don't see how it provides much value during development when it wouldn't actually confirm what you set is actually working with real physical hardware. It would only confirm that what you set changes the fake stream.

[adamaji]

There's some value in verifying that e.g., our application has called SetWobbleRate, but yes there's not much difference on the simulator side between seeing the stream change or seeing that a some internal variable has changed. I also think the demo example you mentioned is a non-negligible motivator, it would be nice to demo our setup without needing hardware.

[lassoan]

The ultrasound simulator device can be used for actual real-time simulation of ultrasound imaging. It uses simulated or real tracking data and STL models of devices, anatomy, etc. The ultrasound noise patterns, beam width artifacts, etc. are not reproduced with high fidelity (the image is typically more clear than in reality), but that may even be useful during development. You can do spatial and temporal calibration, volume reconstruction, etc. on the virtual probe and in silico calibration or anatomical phantoms. It is all there in Plus, it is just a matter of putting together the appropriate Plus configuration file.

[jamesobutler]

I think @adamaji’s primary motivation for the virtual ultrasound device was to be able to test classes like vtkPlusCapistranoVideo source class specifically, but without the requirement of Capistrano hardware. To be able to test if that class and it’s class variables were functioning correctly.