merge function results in unnecessarily huge output

[HassanMasoomi]

Hi,

I have several tiles and I need to merge them; but using terra's merge function will result in a ridiculously huge output file while using raster's merge function behaves very well and the output makes sense. In my real case, the output should be around 7GB but when I use terra's merge function, it results in an output file of ~120GB!!!

I managed to provide a reproducible example here. In this example, I have 80 tiles of ~6.5MB (in total ~520MB). Using terra, the merged file is ~4GB (!!!) while using raster results in a merged file of ~850MB (which is what expected).

Interestingly, if I just rewrite the result from terra, it will be ~850MB which totally would makes sense.

library(terra)
# terra 1.7.55

# creating 80 tiles of 1x1 out of 121 tiles for the region
n <- 80
allComb <- expand.grid(lon = seq(-100, -90, by = 1), lat = seq(40, 50, by = 1))
allComb <- allComb[sample.int(nrow(allComb), n),]
for (i in 1:n) {
  writeRaster(rast(xmin=allComb[i, "lon"], xmax=allComb[i, "lon"]+1, 
                   ymin=allComb[i, "lat"], ymax=allComb[i, "lat"]+1, res=0.0001, vals=i), 
              paste0("part_", i, ".tif"), overwrite = TRUE)
}

# merge tiles using terra
allParts <- list.files(pattern = "^part_.+\\.tif", full.names = TRUE)
all_terra <- merge(sprc(allParts), first = TRUE, na.rm = TRUE, 
                   filename = "all_terra.tif", overwrite = TRUE)

# merge tiles using raster
allParts <- list.files(pattern = "^part_.+\\.tif", full.names = TRUE)
allParts <- lapply(allParts, raster::raster)
allParts$filename <- "all_raster.tif"
all_raster <- do.call(raster::merge, allParts)

# rewrite the terra output
writeRaster(all_terra, filename = "all_terra_rewrite.tif", overwrite = TRUE)

[rhijmans]

I see

file.info("all_terra.tif")$size / 1024^2
#[1] 901.186
file.info("all_terra_rewrite.tif")$size / 1024^2
#[1] 261.0031

describe("all_terra.tif")[c(37, 45)]
#[1] "  COMPRESSION=LZW"                                 
#[2] "Band 1 Block=1100x1 Type=Float32, ColorInterp=Gray"
describe("all_terra_rewrite.tif")[c(37, 45)]
#[1] "  COMPRESSION=LZW"                                 
#[2] "Band 1 Block=1100x1 Type=Float32, ColorInterp=Gray"

It seems that the compression is not working very well when using merge the way that terra implements it.

[rhijmans]

I have added two alternative methods ("algo=2" and "algo=3") that give much better compression, but in my tests, that comes at the expense of speed.

If the merged raster is not a final output, another option is to use "algo=3" with no output filename. That returns a virtually merged raster (no new file) that can be used in subsequent computation.

library(terra)
x <- rast(xmin=-110, xmax=-80, ymin=40, ymax=70, res=1, vals=1)
y <- rast(xmin=-85, xmax=-55, ymax=60, ymin=30, res=1, vals=2)
z <- rast(xmin=-60, xmax=-30, ymax=50, ymin=20, res=1, vals=3)
s <- sprc(x, y, z)

b1 <- merge(s, algo=1, filename="merge1.tif", overwrite=TRUE)
b2 <- merge(s, algo=2, filename="merge2.tif", overwrite=TRUE)
b3 <- merge(s, algo=3, filename="merge3.tif", overwrite=TRUE)
panel(c(b1, b2, b3))