Terrain generation

Terrain generation

To create realistic environments, a skybox is not enough, and a terrain must be added. The main difficulty of this part is to match the terrain with the skybox, in order to get a good rendering. As we consider that the skybox is already good, it can be used to generate a terrain.

Image-to-terrain model

As far as I know in (2024-09), there is no good open-source model to generate a terrain from the image. This solution would be the more direct, but we will have to create our own pipeline.

Matching 2D image and 3D terrain: depth generation

A terrain is essentially a 3D surface. A simple transition from an image to a 3D environment is to use a depth generation model ( Models on Hugging Face ). The details on this technique are documented at RGBD image generation. From now on, we consider that we have RGBD (D for depth) data for the scene.

Terrain segmentation

How to separate the voxels that represent the terrain from the other? This is a segmentation task. The segmentation is explained at RGBD image#Segmentation.

From RGBD image to RGBH terrain

A terrain is often represented as a combination of a texture and an heightmap. Both elements are viewed from the top with an orthographic projection, while the RGBD image uses a perspective projection. To change between projection systems, we use the image gradient to complete "the shape of the visible terrain". In the future, a real heightmap would be preferable.

When the terrain is segmented (and completed), it is still a RGBD segment. The projection change is the best way of generating a real terrain, but this task is not finished yet.