US Simulation from CT

[Kevin-Veronneau]

I'm currently trying to decide the feasibility of using PLUS for US simulation from a set of CT Images I already have. I currently have a program built with VTK 9.1 and Qt 6.3 which uses the trakSTAR 2 with a needle sensor, sensor for locating the phantom, and a sensor to simulate an ultrasound probe. I've got most of the application working and it will display the correct CT Image based on where the US sensor is. It would be nice to have the image look more like and ultrasound image. I have built a model of the phantom from the CT files using vtkDICOMImageReader and am wondering how feasible it would be to use PLUS in the pipeline to simulate the US.

I tried building the PlusBuild branch which unsurprisingly didn't work since I'm using VS2019. I've been trying to read the documentation and discussions to try figuring this out. I'm unsure if I would just be able to use the files from https://github.com/PlusToolkit/PlusLib/tree/master/src/PlusUsSimulator to do this, or if I need to build PlusBuild to get it working at all.

Thanks in advance, Kevin

[ungi]

You can set up the US simulation feature of PLUS. It's somewhat complicated to figure out all transformations correctly, but it's working. That is a stable solution to generate images. For this, you need to segment all anatomy from CT to have 3D models of all organs and tissues. You will need to set up acoustic properties for each tissue. And the images will still look very artificial. But good for training.

If I had to start over today, I would use a CGAN neural network (pix2pix). If you have access to registered CT and US scans, you can train pix2pix with that. If not, you could still do segmentation-to-US. It works surprisingly well, and it generates US images that are almost impossible to tell apart from real US images.

If you choose to develop a new method, the general trend is that we don't add new algorithms to PLUS. 3D Slicer has become much better in the past years. You could just send tracked US images from PLUS to Slicer, and implement any new simulation algorithm in Slicer.

[lassoan]

I agree that neural network based tools can generate realistic looking images and maybe they are even faster on a strong GPU than mesh-based methods, and it may be advantageous that you generate images directly from CT image (without segmentation). There is still some hype around neural networks, therefore it may be easier to get funding if you utilize such methods. Mesh-based ultrasound simulation still has a number of significant advantages, too (see below). So, it depends on your end goal, which approach is more suitable for you.

The main advantage of mesh-based ultrasound simulation that introducing motion, deformation, adding accurate simulation of moving tools is very simple, inexpensive, real-time operation (while warping a 3D volume for image-based simulation is an expensive operation). Mesh-based simulated images also look simpler (which may be and advantage or a disadvantage - depending on the training goal) and you have full control over what appear exactly how. You can generate images in a an position and angle - you don't need to collect ultrasound images from similar angles; in fact you don't need to collect any ultrasound training samples at all. It is also an advantage that it runs on any computer, any laptop, in real-time, while for any deep-learning based tools you need a PC or a clunky gaming laptop.